Eye mounted intraocular displays and systems

ABSTRACT

A display device is mounted on and/or inside the eye. The eye mounted display contains multiple sub-displays, each of which projects light to different retinal positions within a portion of the retina corresponding to the sub-display. The projected light propagates through the pupil but does not fill the entire pupil. In this way, multiple sub-displays can project their light onto the relevant portion of the retina. Moving from the pupil to the cornea, the projection of the pupil onto the cornea will be referred to as the corneal aperture. The projected light propagates through less than the full corneal aperture. The sub-displays use spatial multiplexing at the corneal surface. Various electronic devices interface to the eye mounted display.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.15/868,981, “Eye Mounted Displays and Systems,” filed Jan. 11, 2018;which is a continuation of U.S. patent application Ser. No. 15/265,697,“Eye Mounted Displays and Systems, with Data Transmission,” filed Sep.14, 2016, now U.S. Pat. No. 9,899,005; which is a continuation of U.S.patent application Ser. No. 14/494,327, “Eye Mounted Displays andSystems Using Eye Mounted Displays,” filed Sep. 23, 2014, now U.S. Pat.No. 9,812,096. U.S. patent application Ser. No. 14/494,327 is acontinuation-in-part of U.S. patent application Ser. No. 14/226,211,“Systems Using Eye Mounted Displays,” filed Mar. 26, 2014, nowabandoned; which is a continuation of U.S. patent application Ser. No.12/359,951, “Systems Using Eye Mounted Displays,” filed Jan. 26, 2009,now U.S. Pat. No. 8,786,675, which claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No. 61/023,073, “EyeMounted Displays,” filed Jan. 23, 2008 and to Provisional PatentApplication Ser. No. 61/023,833, “Systems using Eye Mounted Displays,”filed Jan. 26, 2008. U.S. patent application Ser. No. 14/494,327 is alsoa continuation-in-part of U.S. patent application Ser. No. 12/359,211,“Eye Mounted Displays,” filed Jan. 23, 2009, now abandoned, which claimspriority under 35 U.S.C. § 119(e) to U.S. Provisional Patent ApplicationSer. No. 61/023,073, “Eye Mounted Displays,” filed Jan. 23, 2008. Thesubject matter of all of the foregoing is incorporated herein byreference in their entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates generally to visual display devices, andelectronics devices coupled to visual display devices. Moreparticularly, it relates to eye mounted displays, electronics devicescoupled to eye mounted displays, and corresponding applications andoptimizations for such devices and displays.

2. Description of Related Art

More and more our technological society relies on visual displaytechnology for work, home internet and email use, and entertainmentapplications: HDTV, video games, portable electronic devices, etc. Thereis a need for improvements in display technologies with respect tospatial resolution, quality, field of view, portability (both size andpower consumption), cost, etc.

However, the current crop of display technologies makes a number oftradeoffs between these goals in order to satisfy a particular marketsegment. For example, direct view color CRTs do not allow directaddressing of individual pixels. Instead, a Gaussian spread out overseveral phosphor dots (pixels) both vertically and horizontally(depending on spot size) results. Direct view LCD panels have generallyreplaced CRTs in most computer display and large segments of the TVdisplay markets, but at the trade-offs of higher cost, temporal lag insequences of images, lower color quality, lower contrast, andlimitations on viewing angles. Display devices with resolutions higherthan the 1920×1024 HDTV standards are now available, but atsubstantially higher cost. The same is true for displays with higherdynamic range or high frame rates. Projection display devices can nowproduce large, bright images, but at substantial costs in lamps andpower consumption. Displays for cell phones, PDAs, handheld games, smallstill and video cameras, etc., must currently seriously compromiseresolution and field of view. Within the specialized market where headmounted display are used, there are still serious limitations inresolution, field of view, undo warping distortion of images, weight,portability, and cost.

The existing technologies for providing direct view visual displaysinclude CRTs, LCDs, OLEDs, LEDs, plasma, SEDs, liquid paper, etc. Theexisting technologies for providing front or rear projection visualdisplays include CRTs, LCDs, DLP™, LCOS, linear MEMs devices, scanninglaser, etc. All these approaches have much higher costs when higherlight output is desired, as is necessary when larger display surfacesare desired, when wider useable viewing angles are desired, for stereodisplay support, etc.

Another general problem with current direct view display technology isthat they are all inherently limited in the perceivable resolution andfield of view that they can provide when embedded in small portableelectronics products. Only in laptop computers (which are quite bulkycompared to cell phones, PDAs, hand held game systems, or small stilland/or video cameras) can one obtain higher resolution and field of viewin exchange for size, weight, cost, battery weight and life time betweencharges. Larger, higher resolution direct view displays are bulky enoughthat they must remain in the same physical location day to day (e.g.,large plasma or LCD display devices).

One problem with current rear projection display technologies is thatthey tend to come in very heavy bulky cases to hold folding mirrors. Andto compromise on power requirement and lamp cost most use display screentechnology that preferentially passes most of the light over a narrowrange of viewing angles.

One problem with current front projection display technology is thatthey take time to set up, usually need a large external screen, andwhile some are small enough to be considered portable, the weightsavings comes at the price of color quality, resolution, and maximumbrightness. Many also have substantial noise generated by their coolingfans.

Current head mounted display technology have limitations with respect toresolution, field of view, image linearity, weight, portability, andcost. They either must make use of display devices designed for otherlarger markets (e.g., LCD devices for video projection), and put up withtheir limitations; or custom display technologies must be developed forwhat is still a very small market. While there have been many innovativeoptical designs for head mounted displays, controlling the light fromthe native display to the device's exit pupil can be result in bulky,heavy optical designs, and rarely can see-through capabilities (foraugmented reality applications, etc.) be achieved. While head mounteddisplays require lower display brightness than direct view or projectiontechnologies, they still require relatively high display brightnessbecause head mounted displays must support a large exit pupil to coverrotations of the eye, and larger stand-off requirements, for example toallow the wearing of prescription glasses under the head mounteddisplay.

Thus, there is a need for new display technologies to overcome theresolution, field of view, power requirements, bulk and weight, lack ofstereo support, frame rate limitations, image linearity, and/or costdrawbacks of present display technologies. Eye mounted displays (EMDs)as described below are a possible solution. Furthermore, it is in manycases advantageous to make the device “eye mounted display systemaware,” in order to allow optimization of the device (and possibly theEMD also) and additionally to provide greatly expanded features overwhat might be possible prior to EMDs.

SUMMARY OF THE INVENTION

The present invention overcomes various limitations of the prior art bymounting the display device on and/or inside the eye, and by couplingother devices to such display devices mounted on and/or inside the eye.The eye mounted display contains multiple sub-displays, each of whichprojects light to different targeted portions of the retinal surface, inthe aggregate forming a virtual display image. These sub-displaysutilize optical properties of the eye to avoid or reduce interferencebetween different sub-displays and, in many cases, also to avoid orreduce interference with the natural vision through the eye.

It is known that retinal receptive fields do not have anything close toconstant area or density across the retina. The receptive fields aremuch more densely packed towards the fovea, and become progressivelyless densely packed as you travel away from the fovea. In another aspectof the invention, the sub-displays generate the “pixel” resolutionrequired by their corresponding targeted retinal regions. Thus, theentire display, made up of all the sub-displays, is a variableresolution display that generates only the resolution that each regionof the eye can actually see, vastly reducing the total number ofindividual “display pixels” required compared to displays of equalresolution and field of view that are not eye mounted. For displays thatare not eye mounted, in order to match the eye's resolution, each pixelon the display must have a resolution sufficient to match the highestfoveal resolution since the viewer may, at some point, view that displaypixel using his fovea. In contrast, pixels in an eye mounted displaythat are viewed by lower resolution off-foveal regions of the retinawill always be viewed by those lower resolution regions and, therefore,can have larger pixels while still matching the eye's resolution. As aresult, a 400,000 pixel eye mounted display using variable resolutioncan cover the same field of view as a fixed external display containingtens of millions of discrete pixels.

Nature produces images on the human eye through interaction of visiblelight wavefronts from the sun with physical objects. Man made displaysproduce images on the human eye either through the direct generation ofvisible light wavefronts (Plasma, CRT, LED, SED, etc.), front or rearprojection onto screens (DMD™, LCOS, LCD, CRT, laser, etc.), orreflection of light (LCD, liquid paper, etc.). However, these displaysall have defects as previously noted. Mounting the display on the headof the viewer (Head Mounted Displays: HMDs) reduces the requiredbrightness, but introduces limits on linearity of optics, resolution,field of view, abilities for “see-through”, weight, cost, etc.

Many of these defects can be cured by mounting a display to and/orwithin the eye itself. For example, FIG. 57, reference 5700, shows arepresentation of a large number of “femto projector” sub-displaysplaced on the surface of the cornea. Because each display resolution ismatched to the corresponding receptor field resolution, a much lowernumber of pixels (400,000) is sufficient to match the field of view ofan equivalent resolution external display (tens of millions of pixels).However, a direct physical implementation of the geometry of FIG. 57 isimpractical. The viewer cannot blink, or rotate his eyes much.

FIGS. 62 and 63 show one solution to this drawback. The projectors ofFIG. 57 have had their optical paths folded such that they lie in avolume thin enough to be contained within a conventional sclera contactlens. The result is a new type of visual display—an Eye Mounted Display(EMD). Together with external free space pixel data transmitters, eyetrackers, power supplies, audio support, etc. which can be mounted in aheadpiece (which can take the form of a pair of glasses), and additionalelectronics to couple with image generators and head trackersub-systems, the result is an Eye Mounted Display System (EMDS), as willbe described in more detail below.

In one embodiment, the eye mounted display is based on a sclera contactlens that is mountable on the eye. The center of the sclera contact lensis occupied by a display capsule that has an anterior shell, a posteriorshell and an interior. The display capsule is mounted in the scleracontact lens so that the anterior shell of the display capsule is flushto an anterior surface of the sclera contact lens. The sub-displays arefemto projectors located in the interior of the display capsule. Thefemto projectors project light through underfilled corneal aperturesthat are substantially non-overlapping. The apertures are underfilled inthe sense that the projected light does not fill the entire pupil. Thisallows all of the femto projectors to project their light through thecommon pupil. After the posterior shell of the display there is a slightair-gap before a prescription hard contact lens (optional) is present.

In addition to the eye mounted display, an exemplary eye mounted displaysystem also includes an eye tracker and a scaler. The eye tracker tracksthe orientation (and possibly also slight positional shifts) of the eye.The digital pixel processing scaler is coupled to the eye mounteddisplay and to the eye tracker. It receives video input and converts it,based in part on the orientation of the eye received from the eyetracker, to a format suitable for projection by the eye mounted display.

In one implementation, the user wears a headpiece. On the headpiece aremounted part of a head tracker, part of an eye tracker and a data linkcomponent. The other part of the head tracker is positioned in anexternal physical frame of reference, and the two parts of the headtracker cooperate to track the position and orientation of the user'shead. The eye mounted display contains the other part of the eyetracker, e.g., fiducial or other marks tracked by a camera mounted onthe headpiece. The combination of the head and eye tracking data can beused to form an absolute transform from the external physical referenceand the position of points of interest on the eye: the cornea, cones onthe retina, etc.

The scaler performs conversion of video from standard or non-standardvideo sources to a retinal based raster based on the absolute transform.The data link component receives the converted video from the scaler andwirelessly transmits it to the headpiece which will pass it on to theeye mounted display. The (usually) planar video inputs may be mapped toplanar virtual displays generated by the eye mounted display, or theymay be mapped to a cylindrical display or to displays of more complexshape.

There are many advantages of eye mounted displays. Depending on theembodiment, some of the advantages can include variable resolutiondisplays where the number of pixels in the display is significantly lessthan prior art non-eye mounted displays for the same effectiveresolution; very low brightness required of the display (literally aslow as a few thousand photons per retinal cone, approximately onemillion times less photons than a 2,000 lumen video projector);extremely small size and inherent portability (e.g. worn as a contactlens, and/or implanted within the eye, etc.); extremely high resolutionand wide field of view; and potentially lower cost compared to the setof multiple displays that can be replaced by one eye mounted display.

Other aspects of the invention include methods corresponding to thedevices and systems described above, and applications for all of theforegoing.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention has other advantages and features which will be morereadily apparent from the following detailed description of theinvention and the appended claims, when taken in conjunction with theaccompanying drawings, in which:

FIG. 1 shows one embodiment of a logical partitioning of an eye mounteddisplay system.

FIG. 2 shows one embodiment of a physical partitioning of an eye mounteddisplay system.

FIG. 3 shows one embodiment of additional electronics in an eye mounteddisplay system.

FIG. 4 shows example inputs and outputs for a scaler black box.

FIG. 5 shows an example portion of a head tracker system: the trackerfame.

FIG. 6 (prior art) shows a computer workstation with a single directview physical LCD display.

FIG. 7 shows an example of a computer work station with a single virtualdisplay that has the same spatial position, orientation, and size as thephysical display of FIG. 6.

FIG. 8 (prior art) shows an example of a computer workstation with sixdirect view physical LCD displays.

FIG. 9 shows an example of a computer work station with a singlecylindrical virtual display that has substantially the same spatialposition, orientation, and size as the array of physical displays shownin FIG. 8.

FIG. 10 shows three example virtual desk screen configurations.

FIG. 11A (prior art) shows how photons in the natural physicalenvironment can result in visual perception: photons from the sunreflect off a point somewhere on a rock cliff and possibly into a human110 observer's eyes.

FIG. 11B (prior art) is a small section of a projection screen where asingle incoming wavefront of light may produce many more possiblereflected point sources that will propagate out from the screen.

FIG. 12 shows a still camera using an EMDS as a viewfinder.

FIG. 13 shows how scaler functionality can be integrated with a cellphone chip.

FIG. 14 shows a still camera using an EMDS as a viewfinder away from thecamera.

FIG. 15 shows a stereo camera using an EMDS as a viewfinder.

FIG. 16 shows an EMDS with an EMDS aware cell phone.

FIG. 17 shows the configuration of FIG. 16 being used for checking emailand surfing the web while waiting at a bus stop.

FIG. 18 shows a pedestrian wearing an EMDS cell phone accessing avirtual map.

FIG. 19 shows an automobile driver wearing an EMDS cell phone accessinga virtual map.

FIG. 20 shows a virtual storefront for passersby who are wearing anEMDS.

FIG. 21 shows a laptop computer using the virtual image of an EMDS asits display.

FIG. 22 shows a tracker frame, stereo speakers, and a rack of HDTV,audio, and EMDS equipment.

FIG. 23 shows a virtual HDTV display in a home.

FIG. 24 shows a virtual stereo HDTV display in a home.

FIG. 25 shows a virtual large screen format 3D display in a home.

FIG. 26 shows a sports application of EMDS in a sporting stadium.

FIG. 27 (prior art) shows the limits on the field of view of the lefteye.

FIG. 28 (prior art) shows the limits on the field of view of the righteye.

FIG. 29 (prior art) shows the limits on the field of view of stereooverlap.

FIG. 30 shows a full spherical immersion 3D display driven by a gameconsole.

FIG. 31 shows a hand-held gaming device working with an EMDS.

FIG. 32 shows a soldier training in a virtual environment.

FIG. 33 shows a soldier training in an EMDS-enhanced environment.

FIG. 34 shows a virtual command, control, and communications room.

FIG. 35 shows a virtual automobile.

FIG. 36 shows the interior of a virtual automobile.

FIG. 37 shows an engineer designing a crank shaft using EMDS-baseddesktop virtual reality.

FIG. 38 shows a virtual 3-way teleconference

FIG. 39 shows a jet engine technician using an augmented reality repairapplication.

FIG. 40 shows a software engineer using an immersive EMDS.

FIG. 41 shows space telepresence control of an external robot.

FIG. 42 is a perspective drawing depicting imaging of a point sourceonto the retina, as seen from the point of view of the point source.

FIG. 43 shows the same situation as FIG. 42, except from a point of viewrotated half way from the location of the point source and head-on tothe face.

FIG. 44 shows the same situation as FIG. 42, except from a point of viewnow looking head-on to the face.

FIG. 45 is a nine cone retina, to be used as a simplified example.

FIG. 46 shows the optical aperture at the surface of the cornea for eachof the nine cones.

FIG. 47 shows how a single display can address three of the nine conesat the same time.

FIG. 48 shows how three displays can address all nine cones at the sametime.

FIG. 49 shows how to generate the desired point source relative angles,and then use a converging lens to convert them to natural expandingspherical wavefronts for reception by the eye/contact lens.

FIG. 50 shows a mirror angled at 45 degrees to fold the display of FIG.49 flat, so as to better fit within the narrow confines of many types ofEMDs, e.g. contact lens based EMDs, intraocular lens based EMDs, etc.;and also shows a simple converging lens.

FIG. 51 shows a single front surface curved mirror that can provide boththe function of the 45°-angled mirror and the converging lens of FIG.50, also eliminating chromatic aberration and fitting into a shorterspace.

FIG. 52 shows an overhead view of the optical components of FIG. 50.

FIG. 53 shows an overhead view of a variation of the optical pipeline ofthe last two figures, but folding the projection path with a frontsurface mirror.

FIG. 54 shows how four femto-displays can form a four times larger areasynthetic aperture.

FIG. 55 shows how an overhead mirror can make a long femto projectormore compactly fit into the area between two parabolic surfaces (such aswithin a contact lens).

FIG. 56 shows an overhead view of an array of femto displays, tiling theretina to be able to produce a complete eye field of view display.

FIG. 57 shows the unfolded lengths of the projection paths.

FIG. 58 shows a human eye optically modeled in the commercial opticalpackage ZMAX.

FIG. 59 shows spot diagrams of the divergence of the optical beams fromdifferent portions of the femto-display surface as produced by ZMAX

FIG. 60 shows a 3D perspective of an assembled contact lens display.

FIG. 61 shows an exploded view of a contact lens display.

FIG. 62 shows one layer of optical routing.

FIG. 63 shows a second layer of optical routing.

FIG. 64 shows a contact lens mounded display 3D perspective view frombelow.

FIG. 65 shows a horizontal slice view of six time steps of an eyeblinking over a sclera contact lens based EMD.

FIG. 66 shows a horizontal slice view of a contact lens based eyemounted display located on top of the cornea.

FIG. 67 shows a horizontal slice view of an eye mounted display locatedwithin the cornea.

FIG. 68 shows a horizontal slice view of an eye mounted display locatedon the posterior of the cornea.

FIG. 69 shows a horizontal slice view of an intraocular lens based eyemounted display implanted within the eye between the cornea and thelens.

FIG. 70 shows a horizontal slice view of an eye mounted display attachedto the front of the lens.

FIG. 71 shows a horizontal slice view of an eye mounted display attachedwithin the lens.

FIG. 72 shows a horizontal slice view of an eye mounted display attachedto the posterior of the lens.

FIG. 73 shows a horizontal slice view of an eye mounted display placedwithin the posterior chamber between the lens and the retina.

FIG. 74 shows a horizontal slice view of an eye mounted display attachedto the retinal surface.

FIG. 75 shows an example headpiece.

FIG. 76 shows an example of headpiece electronics at a logical level.

FIG. 77 shows an example headpiece from the back side.

FIG. 78 shows an overhead view of an example of electronics contained ina contact lens display capsule.

FIG. 79 shows a block diagram of an example IC internal to the contactlens display capsule.

FIG. 80 shows an example driver chip for a UV-LED bar.

FIG. 81 shows a horizontal cross section of the light creation portionof a femto projector, in this case the phosphor is illuminated frombehind.

FIG. 82 shows a three dimensional perspective view of the light creationportion of a femto projector, in this case the phosphor is illuminatedfrom behind.

FIG. 83 shows a horizontal cross section of the light creation portionof a femto projector, in this case the phosphor is illuminated from thefront.

FIG. 84 shows a three dimensional perspective view of the light creationportion of a femto projector, in this case the phosphor is illuminatedfrom the front.

FIG. 85 shows an overhead view of a contact lens display with largerthan minimal required exit apertures for the femto-displays.

FIG. 86 shows a point source emitting spherical wavefronts of visiblefrequency electromagnetic radiation, and what happens to the portions ofthe wavefronts that encounters the human eye.

FIG. 87 shows more detail on wavefront changes inside the eye of FIG.86.

FIG. 88 is a modification of FIG. 86, in which wavefront portions aredrawn as dotted, dashed, or solid, depending on how their futureencounter with the human eye will go.

FIG. 89 is a modification of FIG. 86, in which only the portions of thewavefronts that will make it to the retina (the solid portions of FIG.88) are shown, along with a thicker line outline showing the envelope ofthis truncated set of wavefronts.

FIG. 90 is a modification of FIG. 89, in which the portions of circulararcs representing the wavefronts at different locations are no longerdrawn, leaving only the envelope to show the limits of all thewavefronts (of FIG. 89).

FIG. 91 (prior art) shows how cone retinal receptive field duals areformed from cone cells at 0° (reference 9110), 0.9° (reference 9120),and 10° (reference 9130) of retinal eccentricity.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Outline

I. Overview II. Some Definitions and Descriptions

II.A. Types of Eye Mounted Displays

II.B. Further Descriptions of Eye Mounted Displays

II.C. Components of an Eye Mounted Display System

III. Making Electronics Devices Eye Mounted Display “Aware”

III.A. Simple Example: An Eye Mounted Display Aware Digital Still Camera

III.B. Modifying the EMDS Scaler Hardware

III.C. Eliminating the Head-Tracker

III.D. EMD Awareness: Resolution

III.E. EMD Awareness: Wide Field of View Aware

III.F. EMD Awareness: Stereo

III.G. EMD Awareness: Head Tracking

III.H. EMD Awareness: Augmented Reality

III.I. EMD Awareness: Virtual Reality

III.J. EMD Awareness: Eye Tracker

III.K EMD Awareness: Additional Object Tracking

III.L. EMD Awareness: Pseudo Cone Pixel Data Stream

IV. Product Classes Combining Electronics Devices and EMDSs

IV.A. EMD Aware Digital and Film Still and Motion Cameras

IV.B. EMD Aware Stereo and Multi-Channel Stereo Still and Motion Cameras

IV.C. EMD Aware Cell Phones and PDAs.

IV.D. EMD Aware Heads Up Display

IV.E. EMD Aware Video Kiosks and Digital Signage

IV.F. EMD Aware Laptop and Palm-top Computer

IV.G. EMD Aware Wearable Computer

IV.H. EMD Aware HDTV Display

IV.I. EMD Aware Day of Release Motion Picture Display

IV.J. EMD Aware 3D HDTV Display

IV.K. EMD Aware Large Screen Format Display, and 3D Display

IV.L. EMD Aware Sports Display

IV.M. EMD Aware Immersive Virtual Reality Display

IV.N. EMD Aware Augmented Reality Display

IV.O. EMD Aware Video Game Software Running on an EMD Non-Aware VideoGame Platform

IV.P. EMD Aware Hardware and Software Video Games on Various Platforms:Hand Held Portable, Portable, Console, Deskside PC

IV.Q. EMD Aware Simulation Systems: Flight, Tank, Dismounted Infantry,Homeland Defense, Firefighting, etc.

IV.R. EMD Aware Real World Systems: Flight, Tank, Dismounted Infantry,Homeland Defense, Firefighting, etc.

IV.S. EMD Aware Real World Systems: Command, Control, and Communications(CCC) center.

IV.T. EMD Aware Full Scale Industrial Design Display

IV.U. EMD Aware Industrial Design Display

IV.V. EMD Aware Telepresence Display for Remote Teleconferencing

IV.W. EMD Aware Augmented Display for Equipment Repair

IV.X. EMD Aware Industrial Virtual Reality Display for SoftwareDevelopment in a Cubicle

IV.Y. EMD Aware Telepresence Display for Remote Medicine, Remote Land,Sea, and Air Vehicles, Space, Planetary Explorations (Moon, Mars, etc.)

V. Eye mounted Displays and Eye mounted Display Systems

V.A. Optical Basis for Eye mounted Displays

V.B A New Approach for Display Technologies

V.C Sub-Displays

V.D Embodiments of Contact Lens Mounted Displays

V.E Internal Electronics of Eye Mounted Display Systems

V.F Systems Aspects for Image Generators and Eye Mounted Displays

V.G Meta-Window Systems for Eye Mounted Displays

V.H Advantages of Eye Mounted Display Systems

I. Overview

FIG. 1 shows an example logical partitioning of an eye mounted displaysystem (EMDS) 105 according to the invention. In this partitioning,there are four elements: the scaler 115, the head tracker 120, the eyetracker 125, and the left and right eye mounted displays (EMDs 130). Forsimplicity, only one EMD 130 is shown in FIG. 1. Two EMDs are generallypreferred but not required. The human user 110, the logical video inputs140, the logical audio outputs 145, and the other I/O 150 are not partof the partitioning.

The EMD system 105 operates as follows. It receives logical video inputs140 as its input, which is to be displayed to the human user 110 via theEMDs 130. In one approach, the EMDs 130 use “femto projectors” (notshown) to project the video on the human retina, thus creating a virtualdisplay image. The scaler 115 receives the video inputs 140 and producesthe appropriate data and commands to drive the EMDs 130. The headtracker 120 and eye tracker 125 provide information about headmovement/position and eye movement/position, so that the informationprovided to the EMDs 130 can be compensated for these factors. Audiooutputs 145 (optional) can also be provided from the logical videoinputs 140. Additional I/O (optional) can also be provided from thelogical I/O 150.

There are many ways in which sub-systems can be configured with an eyemounted display(s) to create embodiments of eye mounted display systems.Which is optimal depends on the application for the EMDS 105, changes intechnology, etc. This disclosure will describe several embodiments,specifically including the one shown in FIG. 2. In this example,portions of the EMDS 105 are worn by a human 110. The overall EMDS 105includes the following subsystems: a daisy-chainable video inputre-sampler subsystem (scalers) 202 through 210, which accept the videoinputs 205 through 208, and 212 through 215, respectively, andadditional I/O (optional) can also be provided from the logical I/O 218through 220; a head tracker subsystem comprised of two parts, 230 and232; an eye tracker subsystem also comprised of two parts, 235 and 238,and a subsystem to transmit in free-space the display information fromthe headpiece to the two EMDs 245 and 248 (left and right eyes).

Portions of these subsystems may be external to the human 110, whileother portions may be worn by the human 110. In this example, the human110 wears a headpiece 222. Much of the data transferred between thesequential scalers 202 through 210 and the headpiece 222, and theheadpiece to the EMDs 245 and 248 is the pseudo cone pixel data stream(PCPDS) 225, to be described in more detail later. The transfer of PCPDSfrom the last scaler 210 to the headpiece 222 can be wired or wireless.If wireless (e.g., the user is un-tethered), then an optional element,the PSPDST pseudo cone pixel data stream transceiver 228 is present.

The head tracker element 120 is partition into two physical components230 and 232, one of which 232 is mounted on the headpiece 222. The otherhead tracker component 230 can be located elsewhere, typically in aknown reference frame so that head movement/position is tracked relativeto the reference frame. This component will be referred to as thetracker frame. The eye tracker element 125 is partitioned into twophysical components 235 and 238. In this example, one of the components238 (not shown) is mounted on the contacts 245 and/or 248, and the othercomponent 235 is mounted on the headpiece 222 to be able to trackmovement of the eye mounted component 238. In this way, eyemovement/position can be tracked relative to the head. The EMDs 130 and135 are implemented as contact lens displays 245 and 248, one worn oneach eye. The audio output an audio output 145 is implemented as anaudio element 250 (e.g., headphone or earbud) that is an optional partof the headpiece 222.

In some cases (to be described later) the head tracker subsystem may notbe required. Each of these subsystems will be described in greaterdetail in the following sections.

An EMDS can be the display portion of a larger electronics system. FIG.3 reference 300 shows the EMDS 310 and other portions of this largerelectronic system that are present. The image generator 320 produces thelogical video inputs 140. This video input could be a still or motionvideo camera, or television receiver or PVR or video disc player (HDTVor otherwise), or a general purpose computer, or a computer game system.This last device, a computer game system could be a general purposecomputer running a video game or 3D simulator, or a video game console,of a handheld video game player, or a cell phone that is running a videogame, etc. The phrase image generator will be used as a higher level ofabstraction phrase for all such devices. Note that traditionaldefinitions of image generator do not always include simple videoreceiver or playback devices. Here, the phrase image generatorexplicitly does include such devices.

Also included in the generic larger electronic system are human inputdevices 340 and non-video output devices 350: audio, vibration, tactile,motion, temperature, olfactory, etc. An important subclass of inputdevices 340 are three dimensional input devices. These can range from asimple 3D (6 degree of freedom) mouse, to a data glove, to a full bodysuit. In many cases, much of the support hardware for such devices issimilar to and potentially shared with the head tracker sub-system 120,thus lowering the cost of supporting these additional human inputdevices.

The phrase scaler, when used in the context of conventional videoprocessing, usually means a processing unit that can convert a videoinput in the format of a rectangular raster of a given height and widthnumber of pixels, with each pixel of a fixed sized, to a video output ofa different format of a rectangular raster of a given height and widthnumber of pixels, with each pixel of a fixed sized. A common example isthe up-conversion of an input NTSC interlaced video stream of 720 by 480(non-square) pixels to an output HDTV 1080i interlaced video stream of1920 by 1080 pixels. However in this disclosure, the term scaler, unlessstated otherwise, will refer to a much more complicated processing unitthat converts incoming video formats, typically of fixed size pixelrasters, to a format suitable for use with the EMDs 130. One exampleformat is a re-sampled and re-filtered non-uniform density video formatwhich will be referred to as the pseudo cone pixel video format, and thesequence of pseudo pixel data will be referred to as the pseudo conepixel data stream. This video format will be described in more detail ina later section. Scalers usually require working storage for the framesof video in. This will be defined as the attached memory sub-system. Thescalers in FIG. 2 implicitly include such memory at this high blocklevel.

FIG. 4, reference 400, shows a particular example scaler “black box”with a specific set of inputs and outputs. The power in is through an ACto DC transformer 405 and DC cable 455, or internal re-chargeablebatteries (not shown) when the scaler is being used in a portableapplication, or power over one or more of the USB connections 435. Thelogical video inputs 205 through 208 are realized through two physicalHDMI inputs 425 and 430. CAT6 physical cables are used to pass thePseudo Cone Pixel Data Stream (PCPDS) from one scaler to another: oneside to/from 410, on the other side from/to 415. Note that while thePCPDS flows only in one direction, the signals carried on the CAT6cables are bi-directional. Other classes of data flow in the opposite orboth directions.

In this example configuration, each scaler box has an input 420 for thehead tracker sub-system, even though typically only one head tracker persystem will be employed. This avoids having to have a separateheadtracker only black-box. Also, while most configurations will haveonly a single physical head tracker reference frame, for coverage over alarger virtual space multiple head tracker units can be used in acellular fashion.

The box supports four USB inputs 435 and four USB outputs 440. These canbe used for supporting keyboard and mice. The system is capable ofperforming KM (keyboard mouse) switching mapping the same keyboard andmouse inputs to any one of a number of computers connected in the videochain. As many modern displays support USB hubs, if the EMDS system isto replace them, it should support the same hub functionality.

Finally, the scaler supports digital optical fiber TOSLINK audio in 445and out 450. This way, the audio from each of several computers attachedcan either have just their audio output switch in or all or some subsetmixed together (remember that audio is also carried by the HDMI links).If a wireless transport of the PCPDS is supported, this functionalitycould be provided via a separate industry standard box, attached to theoutput CAT6 410 of the last scaler in the line. The scaler may be usingonly the lower layers of the Ethernet data transmission protocol for thetransport of the PCPDS and other data, but it preferably follows thespecifications far enough to allow use of common Ethernet switchers andfree space transceivers. The scaler black box shown in FIG. 4 is merelyan example, representing specific I/O choices for sake of providing aconcrete example.

One example of the head tracker component 230, the tracker frame, isshown in detail in FIG. 5, reference 500. Reference 510 is the physicaltracker body, which may be in the form of a x-y-z set of sticks, but notalways. At each of the three ends of this tracker frame, there areactive electronics 530, 540, and 550. The active electronics might onlyinclude the simplest of timing and sensor I/O capabilities. Thecomputation to turn the sensed signals into transform matrices typicallywould not be included in the tracker frame. Instead, the nearly rawsensor inputs would be passed down the data link, via cable 520 in thisexample. The number crunching on the data will be performed elsewhere inthe EMDS. For example, this computation could take place within one ormore of the embedded DSP elements on the headpiece electronics chip.

To put all this and what follows in context, two examples of pre-EMDSdisplays and the EMDSs that replace are described below.

FIG. 6, reference 600, shows a typical work cubicle 610 with a desk 620,chair 630, computer with integral image generator (e.g., a graphicscard) 640, keyboard 650, mouse 660, and a traditional direct view LCDdisplay 670. The next figure shows what an Eye Mounted Display Systemcan do. In FIG. 7, reference 700, everything is the same as in FIG. 6except the user is wearing an EMDS headpiece 222, a wireless videotransceiver the PCPDST 710 has been added, and the physical LCD display670 is replaced by a virtual display 730 of otherwise the samecharacteristics. One other change is the fabric walls of the cubicle 610are preferably a dark black fabric and the top of the desktop is alsopreferably made of a black material. This will increase the contrast ofthe virtual images against the physical world, without the need foroverly low ambient lighting or overly dark shades on the headpiece.

A more interesting example is when more money has been invested in LCDdisplays. FIG. 8, reference 800, shows a work cubicle 610 with not one,but six physical LCD displays: 810, 820, 830, 840, 850, and 860. Now the(almost) same EMDS of FIG. 7 can take in the six video outputs that inFIG. 8 were connected to the six physical LCD displays, and instead theyare connected to six “scaler” virtual video inputs. FIG. 9, reference900, shows the results: six virtual screens placed on a continuouscylindrical display 910, otherwise delivering the same visualinformation as the set-up in FIG. 8 does, but much more flexibly, andpotentially at a lower cost. Note: rather than just projecting to acylinder, the projected surface can be a more general elispse.

More complex virtual display surfaces are possible and contemplated.FIG. 10 shows such additional types. The display 1005 has a flat desksurface 1020 as well as a flat (in the vertical) portion of the virtualdisplay 1010, connected via a ninety degree circular section 1015 of thevirtual display. Assuming circular curving, a three dimensionalperspective view of this display is shown as reference 1025. The display1030 has a flat desk surface 1040 as well as a parabolic (in thevertical) portion of the virtual display 1035, directly connected.Assuming circular curving, a three dimensional perspective view of thisdisplay is shown as reference 1045. The display 1050 is more appropriatefor standing rather than seated use; it has a small tilted desk surface1060 as well as a parabolic (in the vertical) portion of the virtualdisplay 1055, directly connected. Assuming circular curving, a threedimensional perspective view of this display is shown as reference 1065.Three of the many ways in which such complex compound surfaces can besupported will be described. One method is for the scaler to directlysupport such compound surfaces. Another method is to dedicate a scalerto each one of the compound surfaces (e.g., 3 or 2 dedicated scalers).Another method is for such surfaces to be directly supported by theexternal image generator.

While the primary application of an EMD is to the human eye, and most ofthis disclosure will assume this as the target user base, an EMD can bemade to work with animals.

II. Some Definitions and Descriptions

II.A. Types of Eye Mounted Displays

An eye mounted display (EMD) is a device that is mounted on the eye(e.g., directly in contact with or embedded within the eye) and projectslight along the optical path of the eye onto the retina to form thevisual sensation of images and/or video. In most eye mounted displays,as the eye makes natural movements, the display's output is locked to,or approximately locked to, the (changing) orientation of the physicaleye. In this way, the projected images will appear to be stationary withrespect to the surrounding environment even if the user turns his heador looks in a different direction. For example, an image that appears tobe four feet directly in front of the user will appear to be four feetto the user's left if the user looks to the right. An example of an EMDis described in further detail in Section V.

An eye mounted display system (EMDS) is a system containing at least oneeye mounted display and that performs any additional sensing and/orprocessing to enable the eye mounted display(s) to present visual datato the eye(s) emulating aspects of the natural visual world, and/oraspects of virtual worlds. An eye mounted display system may also allowexisting standard or custom video formats to be directly accepted fordisplay. Significantly, in some implementations multiple such videoinputs can be simultaneously accepted and displayed.

One example is the emulation of most present external direct viewdisplay devices (such as CRTs, LCDs, plasma panels, OLEDs, etc.) andfront and rear view projection display devices (such as DLP′, LCD, LCOS,scanning laser, etc.) In this case, an EMDS 105 could take “standard”video data streams, and process them for display on a pair of eyemounted displays (one for each eye) to produce a virtual display surfacethat appears fixed in space. Just as with most present external displaydevices, an industry standard cable, carrying video frames in someindustry standard video format, is physically plugged into an industrystandard input socket on some portion of the EMDS 105, resulting in theuser perceiving a display (controlled emission of photons) of the videoframes at a particular (changeable) physical position in space.

One advantage of eye mounted display systems compared to existingdevices is that there is no bulky external physical device emitting thephotons. In addition, a large number of separate video inputs can bedisplayed at the same time on the same device. Also, EMDS 105 can beconstructed with inherent variable resolution matching that of the eye,resulting in a significant reduction in the number of display elements,and also potentially external to the EMDS computation of displayelements. Furthermore, in embodiments of eye mounted display systemsthat are implemented with high accuracy, they can produce imagery at thehuman eye's native resolution limits.

Not only can eye mounted display systems potentially replace existingdisplay devices, because multiple video feeds can be accepted anddisplayed simultaneously (in different or overlapping regions of space),a single eye mounted display system could conceivably simultaneouslyreplace several display devices. Furthermore, because eye mounteddisplay systems are inherently portable; a person wearing a single eyemounted display system could use that system to replace display devicesat a number of different fixed locations (home, office, train, etc.).

Eye mounted displays can be further classified as follows.

Cornea Mounted Displays (CMDs). Within this class, the display could bemounted just above the cornea, allowing an air interface between thedisplay and the cornea. Alternately, the display could be mounted on topof the tear layer of the cornea, much as current contact lenses are. Forexample, see FIG. 66. In yet another approach, the display could bemounted directly on top of the cornea (but then would have to addressthe issue of providing the biological materials to maintain the corneacells). In yet other approaches, the display could be mounted inside ofor in place of the cornea (e.g., FIG. 67), or to or on the back of thecornea (e.g., FIG. 68).

Contact Lens Mounted Displays (CMDs). In this class of Cornea MountedDisplays, the display structure would include any of the many differentcurrent and future types of contact lenses, with appropriatemodifications to include the display. Examples are shown in FIGS. 60 and61.

Inter-ocular Mounted Displays (IOMDs). In this class, the eye mounteddisplay could be mounted within the aqueous humor, between the corneaand the crystalline lens, just as present “inter-ocular” lenses are(e.g., FIG. 69).

Lens Mounted Displays (LMDs). Just as an eye mounted display could bemounted in front, inside, behind, or in place of the cornea, insteadthese options could be applied to the lens, creating several moreclasses of embodiments. See FIGS. 70, 71, and 72. Replacing the lenswith a LMD would likely be surgically very similar to current cataractsolutions.

Posterior Chamber Displays. FIG. 73 shows a display which has beenplaced within the posterior chamber 1345, between the lens and theretina 1360.

Retina Mounted Displays (RMDs). In this class, the eye mounted displaycould be mounted on the surface of the retina itself (e.g., FIG. 74). Inthis particular case, fewer optical components typically are required.The display pixels (or similar objects) could be placed right above thecones (and/or rods) to be displayed to. However, the display must beable to be fabricated as a doubly curved object (e.g. a portion of asphere).

Relative Size of the Eye. Like other parts of the human body, thediameter of the human eye varies between individuals. Specifically foradults, the variance is a Gaussian distribution with a standarddeviation of ±1 mm about 24 mm, and most other anatomical parts of theeye generally scale with the diameter. Most of the literature implicitlyor explicitly assumes an eye diameter of 24 mm, though sometimes adifferent diameter is given. Some types of data, such as angularmeasurements, are implicitly relative, and thus the size of the eye doesnot matter. But other measurements, such as feature sizes on the retinalsurface, or the size of the cornea, or the size of the pupil, do dependon the size of the eye in question. So while this document forsimplicity follows the convention of a default 24 mm diameter eye, eyemounted displays could be made available in a range of sizes in order toaccomplish better fit and function for the majority of the populace.

II.B. Further Descriptions of Eye Mounted Displays

EMDs in Both Eyes. In the general case, for a particular user, eyemounted displays would be mounted on or in both eyes. This eliminates(or greatly reduces) binocular rivalry, increases perceptual resolution,and allows for display of stereo images. There also is a physicalredundancy factor. That does not mean that just a single eye mounteddisplay might be used in special cases: people with only one functionaleye, some patients with strabismus and in certain special applicationswhere display in only one eye is sufficient. The discussion below isgenerally focused on how to couple a display to a single eye. This isjust for simplicity of exposition. Nothing in that description should beconstrued to mean that the most typical application would not becoupling displays to both eyes.

Femto projectors. There are many different ways that the lightgenerating component of an eye mounted display can control the emissionof photon waterfronts that will focus on or about a particularphotoreceptor of the eye (rods or cones). Many of these, if looked at ina certain way, roughly resemble various forms of video projectors,although at a vastly smaller scale. Also, such photon emittingsub-systems usually will not be able to address the entire retina. Manyinstances of them may be present in a single eye mounted display. Tohave a generic and consistent name for this entire class of photonemitters, the term “femto projectors” will be used. Femto, in this case,is not meant to indicate femto-technology, which is defined as havingindividual components in the femto-meter size range. Rather, the termfemto projector is meant to differentiate such tiny projectors fromsmall projectors currently called “pico projectors,” “nano projectors”;the large “micro projectors”; and their larger cousins—just projectors.

Pseudo Cone Pixels. An EMD contains internal light emitting regions thatwill be defined here as pseudo-cone pixels. Each pseudo cone pixel, whenemitting light, will cause a spot of light to excite some specific(after calibration) (possibly extended) point on the user's 110 physicalretina. In general these pseudo cone pixels do not correspond exactly tothe position and size of specific physical cones on the user's retina,but can be thought of as approximately doing that. Specifically, pseudocone pixels projecting into the highest resolution central fovealportion of the retina may be somewhat larger than the actual cone cells.The lattice of the pseudo cone pixels (for example, an irregularhexagonal lattice) will not exactly match that of the physical cones,and in the periphery of the retina, pseudo cone pixels are sized toresemble the locked together sets of cones that make up the centralportion of peripheral visual receptive fields.

However, for the computational task of converting “standard” video inputinto video data for non-uniformly spaced and sized pseudo cone pixels onan EMD, we can concentrate on the pseudo cone pixels as the target“pixels,” and ignore the actual physical retinal cones (or rods). It islikely that future versions of the technology will allow pseudo conepixels to be manufactured or configured to more exactly match aparticular individual's retinal cone and receptive field lattice. Whilesuch systems should provide some incremental additional improvement inuser 110 perceived resolution, such enhanced systems otherwise will beconstructed quite similar to the systems described here.

Pseudo-Cone Pixel Shape. On the femto projectors on the EMD, oneembodiment of the pseudo cone pixels could be hexagonal in shape.Hexagons are already more closely approximated as circles than assquares (in contrast to more traditional “square” pixels). However thehexagon spread function of light by the time that the pixels is imagedon the retina will be close to both the optical blur limit, as well asthe diffraction limit (at least near the fovea). The end effect is thatthe hexagons will be distorted into very nearly circular shapes. This isimportant, because as various graphics and image processing functionsare considered, they must usually think of pseudo cone pixels ascircular, rather than square.

One must also take care with phrases like “imaged onto the surface ofthe retina.” In the periphery, shapes imaged onto a theoretical sphererepresenting the surface of the retina will be quite distorted (due tothe high angle of incidence), but the cones (and rods) of the retina“fix” this problem by tilting by quite a number of degrees to point atthe output pupil of the lens. Thus the “real” imaging surface of theretina is quite different than a simple spherical approximation. Withinthe art described here, these more accurate effects are understood, andtaken into account where appropriate. Thus, phrases like “the surface ofthe retina” are to be understood as meaning the more complex “real”imaging surface defined by the orientations of the light sensors on theretina.

One could also take into account the effect that as pixels are presentedto higher and higher eccentricities, the light enters the cornea athigher and higher angles tilted away from the local normal to thesurface of the cornea (as described in greater detail elsewhere in thisdocument). While in general this extra tilt will help to keep pseudocone pixels imaged onto the retina close to uniformly circular in shape,pseudo cone pixels at the extreme ends of the femto projector can becomeslightly elliptical when imaged onto the surface of the retina. Whileslight distortions usually can be ignored, at some point the retinalshape of pseudo cone pixels should be modeled as elliptical (or otherdistorted shapes). Fortunately the elliptical ratio is constant, and canbe computed beforehand, or in some cases is a simple function of lensfocus (which can be indirectly determined by the relative vergence inthe orientations of the two eyes). In some of the processing steps to bedescribed in following passages, this complication will at first beignored, and then addressed once the full concept has been developed.

Pseudo Cone Pixel Data Steam, Frame of Pseudo Cone pixel data. Thesequence of pseudo cone pixel data that is transmitted between scalerunits and between the last scalar and the headpiece is referred to asthe pseudo cone pixel data stream. Pseudo cone pixel data streams aresplit up temporally into separate video frame of pseudo cone pixel data.All the pseudo cone pixel data contained in a single video frame of suchdata being sent to the headpiece for display on the EMD is referred toas one frame of pseudo cone pixel data.

Pseudo Cone Pixel Video Frame Format, Pseudo Cone Pixel Descriptors. Aframe of pseudo cone pixel data has a pre-defined fixed sequence ofpseudo cone pixel targets on the set of femto projectors that actuallydisplay the data. Because all the (on the order of 40 to 80) femtoprojectors will be operating in parallel, the pseudo cone pixel videoformat cannot sequentially send the entire pseudo cone pixel datacontents for one femto projector before sending any data to any otherfemto projectors. The constraints mean that pseudo cone pixel data fordifferent femto projectors must be interleaved together in the pseudocone pixel video format. This interleaving does not have to be on anindividual femto projector basis, but it can. There is enough FIFOstorage within the various processing elements that various forms ofre-ordering are possible.

All the scalers fetch from their attached storage a video frame worthsequence of pseudo cone pixel descriptors. Each descriptor contains thegeometric and other data that defines them: normal vector to its center,its normalized radius, its color, normalization gain and offset of theparticular femto projector pixel it is targeted to, its femto pixelprojector, and any femto projector edge feathering for seaming togetherwith another neighboring femto projector. This is only one examplecollection of the contents of pseudo cone pixel descriptors; othercollections and ordering within the video stream are contemplated andpossible.

Each scalar will accept a stream of pseudo pixel data from the scalerbehind it, except for the first, which will generate such a streaminternally based on the pseudo cone pixel descriptors fetched from theattached storage, and send it on to the next. Depending on the physicalworld relative position and orientation associated with the frame ofvideo input to a particular scalar, the scalar will contribute data onlyto a sub-set of all of the pseudo cone that pass through it. For thisactive subset, and given the internally fetched pseudo cone pixeldescriptor, the scaler will generate a pseudo cone pixel value fromcontents its frame of input video. This data may replace thecorresponding data for the same pseudo cone pixel destine for the samefemto projector pixel, or let the input override the internallygenerated pseudo cone pixel data, or a more complex merge of the twovalues. In some simple cases of the edges of the rectangle that is theoutput virtual video screen, the merge function may be simple addition.If multiple layers of virtual video screens allowed to obscure portionsof others, then an even more complex merge function must take placewhen, for example, one screen partially obscures another. In its mostgeneral form, merges between different pseudo cone pixels with sametarget cannot be performed until all of such pseudo cone pixels arepresent. One way to accomplish this is to leave in the stream bothpseudo cone pixels, plus any partial pixel coverage information. Thiswill require inserting into the pseudo cone pixel data stream more thanone data frame for a single femto projector pixel pseudo cone pixeltarget; the number of pseudo cone pixels data frames that have to betaken up by these two will be at least two, and possibly more. In fact,as this un-resolved data merge propagates though the scalers, additionalactive pseudo cone pixels addressing the same target may be encountered,and the result will be a further enlarging of the data frames dedicatedto the same target.

Will this enlarging of the data stream result in possible dataunder-runs to the EMD? Because of the FIFOs all over the EMDS 105, andbecause the scalars have 10% or more processing power available thanotherwise needed, and because an upper limit on doubled and more pseudocone pixels that may partially cover another can be computed, the“surge” in data for one target can be absorbed without compromising thedata rate to the pseudo cone pixels. The computation to be performed isto sort out all the partial pixel coverage claimed on this pixel, andthen merge together, in proportion to its coverage, all such than havenot been totally obscured by another. This operation is the same or verysimilar to the operation of computing the continuation of variouspolygons in know sort order for antialiasing in the computer graphicsliterature. While many other methods are possible, one convent one is tolet the last scalar in the chain perform this merging operation. Thenthe output from the last scalar to the headpiece will be free of anyduplicate (or more) pseudo cone pixels. NOTE: each pseudo conedescriptor included a gain and offset for its target femto projectorpixel. The question is, where should the normalization process occur?The most bandwidth preserving place is within the scalar as the rest ofthe pixel value is computed. Another place is in the last scaler in thechain; this might result in slightly improved numeric output values.

II.C. Components of an Eye Mounted Display System

Eye mounted Display System. An eye mounted display system (EMDS) 105usually will include at least three components: the eye mounted display(EMD) itself, an eye tracking component that provides accurate real-timedata on the current orientation and direction of motion of the eye, anda head tracking component that provides accurate real-time data on thecurrent orientation and direction of motion of the head (or technically,the headpiece attached to the head) relative to some physical worldreference coordinate frame 230. There are some practical applications ofEMDs that do not require the head tracking component. However, there arevery few applications of an EMD that will work well without the eyetracking component. The eye mounted display system may also includeother components, including possibly some or all of the following:

Eye Tracker. Typically, an EMDS 105 will know to high accuracy theorientation of the eye(s) relative to the head at all times. Severaltypes of devices can provide such tracking. For the special case ofcornea mounted displays fixed in position relative to the cornea, theproblem devolves to the much simpler problem of tracking the orientation(and movement direction and velocity) of the cornea display. Specialfiducial marks on the surface of the cornea mounted display can makethis a relatively simple problem to solve. Other types of eye mounteddisplays may be amenable to different solutions to the problem oftracking the orientation of the eye to sufficient accuracy.

To generate the proper image to be displayed by an eye mounted display,the image formation preferably takes into account the current positionand/or orientation of the eye relative to the head and/or the outsideenvironment. Technically, eye orientation sensors typically will tellyou where the eye was, not where it is now, let alone where it will beby the time the image is displayed to it. Thus it is desirable to trackthe eye's orientation at a rate several times faster than the displayupdate rate, to allow accurate computation of the recent past rotationaldirection and velocity of the eye. This can be used as a predictor ofwhere the eye will have rotated to by the time the image is displayed toit.

This same high sample rate time sequence orientation information aboutthe eye can also be used to determine which of several different typesof eye motion is in progress: saccades, drifts, micro saccades, trackingmotion, vergence motion (by combining the rotation information from theother eye), etc. Tremor motion during drifts is likely fine enough tonot be sense-able or to make much difference in the display contents.However, if it can be sensed, it can be used in determining fineorientation of the eye, if needed. While not technically an eye motion,many eye trackers 125 can usually also correctly detect eye blinks. Asduring saccades, the eye is “blind” during many of these motions, and inthese cases no image need be computed or displayed. After any motionthat shuts down visual input to the brain ends, there is anapproximately 100 millisecond additional period in which visual input isstill not processed. This allows EMDS 105 that have their own latencytime to determine where the eye is now (e.g., that the motion or blinkhas finished), start computing the correct image to be displayed, andtransfer that image to the EMD and display (emit photons) before the eyestarts seeing again.

The eye, as a sphere, has three independent degrees of freedom relativeto its socket, requiring its orientation to be described by threeindependent numbers. In many cases, using an appropriate representationof orientation, the eye only uses two of these degrees of freedom, asdescribed by “Listing's Law” but the law varies with vergence. Also,during pursuit motions, the eye ignores Listing's Law to keep the targetcentered in sight. Thus in general, an eye tracker 125 preferably wouldsense all three possible independent dimensions of orientations of theeye, not just two. However, the orientational deviations from Listing'sLaw are known to be within a specific small range, and an eye trackersystem can take advantage of these limits.

The eye motion information is also needed to correctly simulate retinalmotion blur, if such blur would have occurred when viewing a physicalobject under similar circumstances. This computation is effected by theduty cycle of “lag” time of the physical display elements, as well asthe current eye motion over the native display “frame” time andhead/body motion over the same period. More details on the requiredcomputation will be described later.

Most eye mounted display applications will require the displayed imageto appear stabilized with respect to the physical space around the user.In such cases, in addition to the rotational position and velocity ofthe eye relative to the head, the position and orientation of the user'shead (and thus body) relative to the physical space around the usershould be known, along with computed temporal derivatives of thesevalues to allow prediction. Some types of eye trackers 125 can give botheye and head tracking 120 information, but usually it is simpler andmore accurate to separate the two functions: an eye orientation tracker,and a head position and orientation tracker, as described in the nextsection.

When trying to determine the orientation of the eye within the angleformed by one foveal cone or less, an accuracy of plus or minus one arcminute or less is preferred in each dimension. Eye mounted displayspotentially allow new inexpensive accurate techniques to be employed toachieve this accuracy.

Head Tracker. Head trackers 120 usually accurately sense six independentspatial degrees of freedom of the human head relative to the physicalspace around the user. One common partitioning of these degrees offreedom is three independent dimensions of position and threeindependent dimensions of orientation. To keep the terminology simple,the discussion that follows will use this common convention, with theunderstanding that there are many other ways to represent spatialinformation about the human head, some of which may have advantages overothers depending on the specific embodiment of the head tracker 120.

Just as with eye trackers 125, most sensed information about the headusually tells one about the past, and so the same sort of super displayframe rate sampling can be employed to compute temporal derivatives ofthe head tracker 120 data (or other data computed from it), which inturn can be used to predict where the future orientation and position ofthe head will be, good for the time frame in which the next image framewill be displayed.

By calibrating the positional and orientation offset from the nativecoordinates of the device attached to the head relative to the center ofthe two (or one) eye(s) of the user, the combined head tracker 120 andeye tracker 125 information describes in physical space the narrow viewfrustum for each cone (or rod) of the retina, within a certain degree oferror. The frustum can be more simply represented by a vector in theviewing direction of the cone (rod), and a subtended half angle of aconical viewing frustum, describing the cone's (rod's) field of view.This information can be used to form the image presented by the eyemounted display(s).

Most existing head tracking technologies do not directly senseorientations, but use three (or more) separate positional measurementsto three (or more) separate points on the headpiece, and thentriangulate (or higher order fit) that data to produce the desiredorientational information. Even the positional measurements are usuallynot made directly. Usually the same target on the headpiece is sensedfrom three (or more) different physical positioned sensors, and thisdata is triangulated (or higher order fit) to produce the desiredpositional information. What is actually sensed varies by device. Somesense the distance between two sub-devices, some sense the orientationbetween two sub-devices, etc. Some devices attempt to sense headorientation directly, but such devices suffer from rapid calibrationdrift (on the order of tenths of seconds), and typically arere-calibrated by a more traditional six degree of freedom head tracker120.

Because of the way the final information is put together (a commonexample is multiple stacked triangulations, not always with very longbase lines), the final accuracy of the head position and orientationdata will usually be less than the native accuracy of the varioussensors used to generate the raw data. How much accuracy is lost (andtherefore how much accuracy is left) can be estimated by performing anumerical analysis of the initial raw accuracy as it propagates throughto the final results. This can also be checked by measuring the actualinformation produced by the head tracker 120 in operation against knownphysical locations and orientations. It is useful to distinguish betweenrelative and absolute (and repeatable) accuracy. Some head trackers 120may give highly accurate position and orientation data relative to thedata it gives for nearby positions and orientations, but the absoluteaccuracy could be off by a much larger amount.

For eye mounted display applications, the orientational accuracy of ahead tracker 120 preferably should be close to the orientationalaccuracy of the eye tracker 125: approximately one arc minute or less.The positional accuracy of the head tracker preferably will be goodenough to not induce shifts in the display image of any more than theangular accuracy. Given that a single foveal cone is on the order of twomicrons across, for a (virtual) object six feet away, a positional errorof not much more than 100 microns is needed to keep the error comparableto a one minute of arc orientational error.

Headpiece. Technically, most head trackers 120 do not track the positionof the head, but rather the position of some device firmly fixed to theuser's head. So long as this device keeps to the same position andorientation with respect to the head to within specified limits, knowingthe position and orientation of the device attached to the head givesaccurate position and orientation information about the head itself.While there are several different possible ways to have devicesphysically attached to the head, for the purposes of exposition andsimplicity, the EMDS 105 described in this document will usually assumean embodiment of a single physical device worn on the head of the user,called the headpiece, upon which many different things may be mounted.The headpiece in most cases does not include the two (one) eye mounteddisplay device(s) mounted to the eye(s), or implanted elsewhere withinthe eye's optical path. Again, this is only one example used forsimplicity of exposition. The same results can be achieved by multipledevices not all attached to each other, or in some cases, just markspainted on the user's head, or nothing at all.

The headpiece could take on many forms. It could look like a traditionalpair of eye glasses (but without any “glass” in the frames), orsomething more minimal, or more complex, or just more stylish.

The devices likely to be attached to the headpiece include thefollowing: elements of the head tracking system (active or passive),elements of the eye tracking system, the device that transmits the imagedata wired or through free space to the EMD proper, the device thatreceives wired or through free space back channel information from theEMD proper, possibly devices that transmit power wired or through freespace to the EMD proper, corded or cordless devices to transmit theimage data from other portions of the EMDS 105 to the device thatforwards the data to the EMD proper. Devices that could be placedelsewhere, but in many cases might be attached to the headpiece includethe following: the computational device that processes raw eye tracking,the computational device that processes raw head tracking data, thecomputational device that processes eye and head track data intocombined positional estimates, orientational estimates, and estimates oftheir first temporal derivatives. Depending on the larger system design,the image data may have one or more of the following operationsperformed on it: decryption, decompression, compression, and encryption.Also, as most new digital video standards also carry high qualitydigital audio data on the same signal, the headpiece could haveprovisions to output analog or digital forms of this data through anaudio output jack. Alternately, the headpiece could have some form ofaudio output (earbuds, headphones, etc) directly built into it.

Transmission of Signals between Components. An eye mounted displaysystem will include a number of sub-systems, which will communicate witheach other. Depending on how the sub-systems are partitioned andconstructed, different methods of communicating data between them areappropriate. In many cases free space communication is not necessary,and physical interconnects (electrical, optical, etc.) are sufficient.In general, wherever possible, industry standard physical layers thatmeet the bandwidth and latency requirements between two sub-systemsshould be used, and the use of corresponding industry standard protocollayers again where possible. One good example is the use of the 10mega-bit, or higher, Ethernet standard. In other cases, sub-systems maybe located so physically close that direct wiring between them ispossible (e.g., on the same PC board).

Finally, when linking one or more components of the EMDS 105 that arenot located on the user, e.g., not being worn, to some part that isbeing worn, it is desirable that a short free space connection beutilized, so that the user does not have to be “tethered.” Currentspread-spectrum short distance wireless interconnects utilizing standardEthernet protocols are one example of existing hardware that meets theun-tethered requirements. In other applications, such as game systems,tethering may be less of a nuisance, worth the cost reduction, and/ortethering of other devices was already required.

Video Input Raster. The physical electrical (or optical or other)transport level of the video to the EMDS 105 may be any of manydifferent standard or proprietary video formats. The most commonconsumer digital video formats today are from the related family ofDVI-I, DVI-D, HDMI, and soon UDI and the new VESA standard. HDMI and UDIalso contain digital audio data, which an EMDS with headphones, earbuds,or other audio output may wish to use. There are also a number ofindustrial digital video formats, including D1 and SDI. The older analogvideo formats include: RGB, YUV, VGA, S-video, NTSC, RS-170, etc.Devices are commonly available to convert the older analog formats intothe newer digital ones. So while a particular EMDS product may haveadditional circuitry for performing some or all of these conversions forthe user, for the purposes of this discussion we will concentrate onwhat happens after the video raster has been converted to, and presentedto the EMDS, as an un-encrypted digital pixel stream. Specificallyconventional issues such as de-interlacing, 2-3 pull-down reversal, andsome forms of video re-sizing and video scaling will also be assumed tohave been performed prior to presentation to the EMDS, or in additionalEMDS pre-processing circuitry that will not be discussed further here.

Different video formats employ different color spaces andrepresentations. A given EMDS 105 component may also employ its ownspecific, and thus not necessarily standard, color space and format. Soin addition to any “standard” color space conversions that may have beenapplied in earlier stages (including brightness, contrast, colortemperature, etc.), an EMDS will usually have to perform an additionalcolor space transform to its native space. In many cases this transformcan simply be folded into a combination transform that already had toexist for conversion of video input from various standard color spaces.Specifically, because of the nature of the computations that will beperformed on the input video data, in the preferred environment theinternal color space for most of the processing will be a linear colorspace. Any non-linearities in the actual pixel display elements areconverted after most of the rest of the processing has been performed.Now, on the one hand, converting to a linear color space requires morebits of representation of pixel color components than non-linear colorspaces. On the other hand, once inside the EMDS, we know the maximumnumber of linear bits that each pixel of the EMD is capable ofdisplaying, and what, if any, dithering is going on. Thus the internallinear color space representation of pixel color components can besafely truncated at some known maximum.

Eye Tracking, Dual Eye Support. In addition to the head trackingcomponent, an EMDS 105 typically also includes an eye trackingcomponent. Note than in some cases, such as a cornea mounted display(CMD), the “eye” tracker 125 may not need to track the eye directly, butcan instead track something directly physically attached to the eye(e.g., the CMD device). Also, while we will focus on the processingneeded to provide data to one eye's EMD, an EMDS will usually supportparallel computation of slightly different data for the EMD in each ofthe two eyes supported. Such stereo display support is important evenwhen viewing mono video sources. Among many other advantages, this willkeep eye fatigue and possible nausea to a minimum. While it is the goalof one embodiment that a single scaler component (described below) willbe able to process and generate output for both eyes in the most complexinput case, so long as provisions are made to deliver input video datato two scaler components in parallel, each handling a single eye each, adoubling of the maximum processing obtainable by a single scalercomponent is easily achieved (at the price of approximately doubling thecost of the scaler element).

Scaler Element, Scaler Component, Scaler Black-Box. In the logicalpartitioning of an eye mounted display into four elements, presented inFIG. 1, one of the logical elements was named the scaler 115.Computations related to the conversion of normal raster video data tothe special display needs of an EMD are performed by this unit.Physically, the scaler element might be physically implemented as asingle integrated circuit chip, perhaps with some DRAM attached, but thescaler element might be implemented as several chips, as eluded to inFIG. 2, in the multiple references 202 through 210, or as portion of alarger chip, as will be discussed later. So without narrowing the scopeof this disclosure, in many examples a scaler component will beone-to-one with a physical integrated circuit chip, plus some attachedDRAM. Because scaler components can be daisy-chained together, in someexamples a collection of scaler components may be referred to as a“scaler black box,” where the logical element scaler may consist of morethan one such black box.

Scaler Component Technical Details. Generally the input to an EMDS 105is some form of rectangular, scan line by scan line sequence of pixeldata, as defined above as the Video Input Raster. However, the type andformat of data that the EMD proper consumes can be quite a bitdifferent. In some embodiments, the EMD consumes a sequence of pseudocone pixel data, usually interleaved so that multiple femto projectorscan be displaying their native format of photon data. While nearly allexisting Video Input Rasters (not compressed video data) are uniform inpixel density (though not always color density), pseudo cone pixels mostcertainly are not. Converting from the standard input formats to thedesired output format is the job of one or more scaler components. Thesecomponents dynamically re-sample and filter the original video data intore-scaled pixels that match the requirement for each output pseudopixel. Indeed, in some embodiments, a portion of the scaler elementinternal data buffers is set aside as storage for a target descriptorfor each pseudo cone pixel to be generated per frame.

How individual components and collections of components are assembled toform a scaler element can be similar to what occurs many times on theother side of the video interface: video cards. Many modern PC videocards have the option of driving two displays at the same time throughtwo separate connectors on the same single card. However, there may be amaximum number of pixels for dual displays that is less per display thanwhat the card can do when driving only a single display. To get higherperformance, a user may prefer that a single graphics card drive only asingle display, or as in several PC gaming cards now, two or even fourgraphics cards can drive just a single display, with not quite linearincreases in delivered graphics performance. The situations forcomponents and collections of components in the scaler element can havesimilar dependencies.

Let us define the smallest unit capable of performing the computation ofa scaler element within a defined set of constraints a scaler component.In many, but not all cases, this may take the form of a single ASIC withother support chips attached, such as DRAM. The scaler element of anEMDS 105 is defined as the entire collection of one or more scalercomponents that perform all the scaler computations for the EMDS. Howmany scaler components will be needed to perform the scaler function foran EMDS will depend on the number of video inputs, the size in pixelsand pixel data rate of each video stream, the form of scaler desired(e.g. projection onto a flat virtual screen vs. projection onto acylindrical virtual screen), type of stereo processing desired, detailsof the EMDs being used, among other factors. In certain special cases nostand-alone scaler element is required at all, either because thefunction has been embedded into another device (such as a cell phone),or the interfacing device is capable of generating correct pseudo conepixel data streams, such as a “pseudo cone pixel aware 3D graphicsrendering engine.”

From a user point of view, there will be one or more types of physicalscaler black boxes available, each with one or more video inputs in oneor more video formats. Multiple such units can be daisy-chainedtogether, before connecting to the free-space or physical cableconnection to the headpiece. These “black boxes” will be differentiatedin the number and type of video inputs on the box, and the limits on thescaler computations that they can perform, as well as the physical powerthat they require. Even for a given unit, the amount of physical powerthat they consume may be variable, depending on the amount of work theyare required to perform. Thus a box that needs to be plugged into a wallwhen working with a complex deskside computer system may only need abattery or power from a USB port when being used with a mobile laptopcomputer. To support such functionality, the ASIC (if that is thetechnology deployed) can have built in the capability to turn offsections of the internal processors when they are not needed, as well asslow down the clock to the powered computations. In this way, twoexpensive ASICS do not have to be constructed. One chip can perform ineach special environment.

Scaler Component Architecture. There are many possible internalarchitectures for the scaler component. One approach is to use a custommicrocodable VLIW SIMD fixed point vector processor. Power can be savedby powering off individual ones of the MD units, and/or lowering theclock frequency to the processor. The microcode is not fixed, but isdownloaded at system initialization time. In this way additionalfeatures can be added, or support of newer model EMDs is possible.

Stereo Support. While the output display is stereo, for the maximumcomfort of the viewer, in most of the cases described here the inputvideo is mono, and the physical display device being emulated is flat.However, with little additional hardware, the systems described here canalso support field sequential stereo or separate left and right eyevideo streams.

Rod Vision. While much of the discussion that follows will be cast interms of controlling light to individual cones of the retina (or in theperiphery, specific neighboring groups of cones), the same technologywill also deliver photons to the more numerous rods of the eye. Thetechniques described below in terms of cones equally apply to rods, onlyso long as lower overall light intensities are involved. A specificexample might be an eye mounted display that is meant to be used withthe user's night vision. Here the display intensity would be kept lowenough to only engage the scotopic rod vision, and would produce a blackand white display. This in fact could just be a “night vision” intensitysetting of an eye mounted display that can also produce brighter imagesfor photopic “daylight” display. Even though there are several timesmore rods than cones (80 to 100 million rods vs. approximately 5 millioncones), the rods tend to group together as larger effective pixel units,and the spatial frequency resolution of scotopic vision is considerablyless that photopic vision. Thus, any eye mounted display that producesanywhere near close to enough spatial resolution for photopic (cone)vision, can also produce more than enough spatial resolution forscotopic (rod) vision.

Safety. EMDs can be see-through, partially see-through, or opaque. Forsafety reasons, in general and consumer applications, it is preferablethat the eye mounted displays be see-through, so that normal vision isnot seriously affected by the eye mounted display. If a truly immersiveapplication is desired, one can put on black out shades. The overallrange of brightness of display of the eye mounted display can also be anissue. With a see-through design, the eye mounted display has to competein brightness (photon count) with the ordinary external world. In adimly lit office or home environment, this is not a hard goal. In directsunlight, eye mounted display intensities of 10,000 times greater wouldbe needed. This is by no means technically impossible, but a competingsafety goal of making it impossible for the eye mounted display to evercause permanent retinal damage may require an artificially limitedmaximum brightness of an eye mounted display. Such a display can stillbe used quite easily in sunlight, for example by wearing fairly darksunglasses, or, more generally, programmable density filters to theexternal world, similar to current variable sunglasses or welding maskwindow technology. This cuts the brightness of the sunlit sceneconsiderably, while not affecting the eye mounted display intensity,because the eye mounted display is “behind” the sunglasses.

See-Through Constraints. Some EMD designs inherently allow forsee-through of normal (standard contact lens corrected, if necessary)vision of the real-world. When the EMDS 105 is off (or showing justblack), the EMD will function purely as a slightly darkening contactlens. Other EMD designs only work as non-see-through. In this instance,the effect is similar to wearing a non-see-through HMD. As the (variabledensity) see-through design is the more general, and can always emulatenon-see through designs by the simple expedient of having the EMDSwearer don a pair of total blackout glasses or goggles, most of thediscussion here will be of the see-through design.

Just because a design is see-through does not automatically mean that itis simple to simultaneously operate in the existing physical world (saya business office) as well as seeing one or more virtual displaysgenerated by an EMDS 105. As discussed elsewhere, a given EMD design maynot be bright enough to compete directly with the brightness of even anormal office environment. One possible compromise is to darken thevariable density shade in the headpiece to view mostly the virtualdisplays, and then un-darken them when needing to interact with the morebrightly lit physical world. The switching from one to the other can becontrolled by the head and eye tracker 125, if necessary, as they knowwhen one is looking at the virtual screens versus the physical world.Thus the switching is seamless. An additional enhancement to allow forvirtual displays to be only as bright as the (partially shaded) physicalworld is to have a region of very dark material (such as black felt)attached to locations in the physical world corresponding to where thevirtual displays are placed. Thus when looking at the virtual displaysthere is no competing light from the physical world, and when looking atthe physical world there is no competing light from the virtual world.

III. Making Electronics Devices Eye Mounted Display “Aware”

Before describing specific product combinations, this section presents anumber of different ways in which an electronic device may wish toemploy EMDS technology. There are many ways in which sub-systems can beconfigured with an eye mounted display(s) to create embodiments of eyemounted display systems. Which is optimal depends on the application forthe EMDS 105, changes in technology, etc. This disclosure will describeseveral embodiments, specifically including the one shown in FIG. 2. Inthis example, portions of the EMDS 105 are worn by a human 110. Theoverall EMDS 105 includes the following subsystems: a daisy-chainablevideo input re-sampler subsystem (scalers) 202 through 210, which acceptthe video inputs 205 through 208, and 212 through 215, respectively, andadditional I/O (optional) can also be provided from the logical I/O 218through 220; a head tracker subsystem comprised of two parts, 230 and232; an eye tracker subsystem also comprised of two parts, 235 and 238,and a subsystem to transmit in free-space the display information fromthe headpiece to the two EMDs 245 and 248 (left and right eyes).

Portions of these subsystems may be external to the human 110, whileother portions may be worn by the human 110. In this example, the human110 wears a headpiece 222. Much of the data transferred between thesequential scalers 202 through 210 and the headpiece 222, and theheadpiece to the EMDs 245 and 248 is the pseudo cone pixel data stream(PCPDS) 225, to be described in more detail later. The transfer of PCPDSfrom the last scaler 210 to the headpiece 222 can be wired or wireless.If wireless (e.g., the user is un-tethered), then an optional element,the PSPDST pseudo cone pixel data stream transceiver 228 is present.

While the primary application of EMDS 105 are to the human eye, and mostof this patent application will assume this as the target user base, anEMDS can be made to work with animals other than man.

III.A. Simple Example: An Eye Mounted Display Aware Digital Still Camera

We start with a simplified example of a digital still camera tointroduce the concept of EMDS awareness. More complex examples will bedescribed in section IV.

Most digital still cameras show a live but low resolution display ofwhat the camera is looking at before the frame is acquired. This lowresolution is due to using small, low resolution LCD (or other) displaydevices, typically fixed to the back of the camera. However, it is alsodue to the processing time it takes to convert what the camera's sensorsees (typically a Bayer array pattern) to an image that can be displayedin an RGB (or similar) format(s). However, a camera that is eye mounteddisplay aware could be generating full camera resolution pixels for thesmall area where the camera user 110 is currently looking, and do lessprocessing at higher visual eccentricities. This situation is shown inFIG. 12, reference 1200, where the photographer 110 wearing an EMDS 105looks “through” the EMDS aware camera 1210 (a specific type of imagegenerator 320) via the EMD's virtual display 1220, which is being usedas a virtual viewfinder for the real-world subjects 1230. The display isthus also demounted from the camera body proper, allowing for a numberof more flexible display effects.

One such situation in shown in FIG. 14, reference 1400, in which thecamera operator 110 is remotely monitoring the shot through a virtualviewfinder 1220, via a physical cable, if necessary, to an EMDS 105 theoperator is wearing, of a group pose of people 1420 that the cameraoperator 110 is in. Another option is for the camera operator to wearthe camera on his head, or over one eye. Note that only minimaladditional circuitry need be added to the camera's internal chip set,because in this special case, we know where not to process highresolution camera data. That is, camera data is processed at resolutionscorresponding to the retinal regions where the data will be displayed.Similar functionality could also be applicable to digital video cameras.

III.B. Modifying the EMDS Scaler Hardware

In the previously described embodiments of eye mounted display systems,a scaler was described, whose input/output function was to take in oneor more video streams, and convert them into a pseudo cone pixel streamfor one or both eyes. This scaler had many possible extra features:seamless edge matching of multiple video streams, projecting onto avirtual display 1220 surface in the shape of a cylinder, proper seamingof one display image in front of another, etc.

For a lower power, lower weight, lower cost in a specific product thatdoes not need all the functionality of a general purpose scalercomputation, simplified scaler components can be designed, and in manycases could be placed directly on one of the special chips that thespecific product already needed for its function.

A common simplification of the scaler computation is to assume thefollowing: there is only a single video stream present; the virtualimage of the video stream is flat in space; the maximum number of sourcepixels is known; and the minimum and maximum subtended field of view ofthe virtual image is known. These simplifications eliminate the need forsupporting curved virtual images, the need for edge seeming or occlusionedges, the need for large image buffers beyond a fixed maximum, the needto triple buffer the image, and sometimes the replacement of the doublebuffer with a single buffer when this will not produce unacceptableimage artifacts for the specific application. Furthermore, the bound onimage size in pixels and extent in degrees places an upper bound on thecomputation rate that the scaler performs, which can allow for a lighterweight scaler sub-system, in some cases on one of the chips that wasalready needed for the primary functionality of the device. Because manylow power, portable target devices already have a built in frame buffer,the primary addition to these devices may be the inclusion of thesimplified scaler element. In some cases where the frame buffer size wasarbitrarily limited by the pixel count of small physical LCD (or other)display devices, adding eye mounted display system awareness is also anopportunity to enlarge the internal frame buffer pixel count.

This can be seen by comparing the on-cell phone chip scaler in FIG. 13,reference 1300, with FIG. 2. In FIG. 2, there are n full functionscalers: scaler 202 through scaler 210. In FIG. 13, there is only onescaler 1330 using only a fraction of the die area on the cell phone chip1310. Furthermore, rather than a separate (and perhaps off-chip)attached memory sub-system for containing a copy of the frame bufferwithin each scaler 202 through scaler 210, the scaler 1330 uses theexisting frame buffer 1320 already present on the cell phone chip 1310.The output of the scaler 1330 is the PCPDS 225. A cell phone chip wasused in this example, but the same approach can be used for many handheld battery powered devices that already present images to its humanuser 110: hand held games, hand held still and/or video cameras, PDAs,electronic books, etc. The existing chip and frame buffer can be usedwith only a small amount of additional circuitry for the scaler 1330 tomake the device eye mounted display aware.

III. C. Eliminating the Head-Tracker

While in more general cases both head tracking and eye tracking may beperformed, some applications may be adequately served without a headtracker 120.

One example could be cameras of all kinds. If the user 110 is holdingthe camera up to his eye(s), or the camera is attached to his head, thenhead tracking per-se is not required because the image input device hasa fixed relationship to the user's physical head.

Other examples include cell phones and PDAs. While the advantages of thedisplay appearing as a stabilized image in physical space might bedesirable, for many simple tasks, having the display in a fixed portionof the user's 110 field of view can be sufficient.

III.D. EMD Awareness: Resolution

Many modern displays have a mechanism that allows sources of displayoutputs to determine what resolutions the display device supports. Thisinformation can be specific specifications sent over a serialidentification protocol or it can be just the device identifying itsmake and model and the source can have an independently loaded table ofinformation on this device. Nowadays, nearly all display technologieshave a fixed pixel resolution for any specific product. This fixedresolution is called the “native” resolution of the display. It ismeasured in the number of pixels wide by the number of pixels high. Inmany cases the display refresh rate may not be continuous, but quantizedinto only a few or even just one update rate.

Note that most CRT's can accept video display signals well above theresolution that would make any difference in the quality of the imagedue to the way their analog interface works. With fixed pixel systems,such as LCD panels, extra computation is performed in real-time todown-scale video formats higher in resolution than the “native”resolution of the device. While there is some cost associated withincluding such circuitry, the legacy ability of CRTs to perform thistask has meant that most fixed pixel displays include this feature. Alsonote that the reverse holds as well. Video inputs lower in resolutionthan the native resolution of the device are up-scaled and displayed. Ifthe device has only one or a few native display refresh rate(s), againcircuitry is usually added to emulate the continuous range of displayupdate rates. Because the generators of video to the display devices aregenerally programmable in the display formats that they can generate,for the best “quality” display, it is common for the user to have thevideo circuitry generate as output the native resolution of the displaydevice.

Because EMDS have a variable native display pixel size, the input videois usually re-scaled to match the more complex “raster” of the EMDS.Because the input video to an EMDS is (almost) always subject toprocessing by the scaler, the EMDS can accept a wide range of videoresolutions. For the ones that it cannot, there are usually outboarddevices that can convert the video signal into one that it can accept.

Looked at in one way, EMDS have no native video resolution. However whenthe effects of the real-time eye tracker are taken into account, it canbe argued that an EMDS has a native resolution of its highest fovealpseudo cone pixels, and a very large number of pixels in width andheight. This can be thought of as the first stage of “EMDS” awareness.Not only can nearly any video resolution be handled, but the virtualphysical size of the display and distance to the display isprogrammable. In some of the simple cases to be described, the imagegenerator just asserts that it is a display of a specific format, e.g.1080p HDTV, or digital IMAX™.

III.E. EMD Awareness: Wide Field of View Aware

Most EMDs preferably support very wide fields of view (100° horizontallyor more). Such capabilities are rarely found with other technologies. Asdescribed elsewhere, fields of view of 65° to 85° start to become veryimmersive. Thus devices can utilize the wide field of view for one orboth of displaying a large amount of data, and causing immersion. Thevertical fields of view are usually smaller, more likely 70° to 80° forfull immersion, as described in the literature.

III.F. EMD Awareness: Stereo

Another level of awareness that an output device can have of an EMDS isto know that the EMDS supports stereo display. There have been twotraditional ways to output the dual images per frame of stereo imageryto a display device. The first is known as “field sequential stereo.”Here, for every frame of display, two fields of video are generated.This is a temporal multiplexing. The first field is a full frame ofdisplay for one eye (usually the left) and the second field is a fullframe of display for the other eye. In an EMDS, the handling of stereoinput is usually performed by the scaler. Normally, the scaler takes ina single field of video per frame, storing it (typically in attachedDRAM), then later re-scaling it by separate left and right eye traversalchannels. To support field sequential stereo input, instead of storingone field per frame in a buffer, store two fields per frame in the framebuffer; then during output traversal, just point the left and right eyescaler elements at the two different buffers, rather than the samebuffer.

The other common stereo video format is to have two separate (butsynchronized in timing) video streams: one for the left eye and one forthe right eye. Once again, this is relatively simple for the scaler tohandle. The brute force solution is to have each video stream go to adifferent portion of the scaler, and then during pseudo cone pixeloutput, only start the left eye traversal in one portion and only startthe right eye output in the other portion. However, if the scalersupports more than one video input per scaler “black box,” then the twoeye video streams can be consumed by one scaler on separate video inputconnectors, storing each port into a separate buffer and then performingthe output processing identically to the method described for fieldsequential processing.

If the input video stream is too much for a single scaler “black box” tohandle, then the work can be divided up between two or more scaler“black boxes” in the same manner as described before for high speedvideo inputs, but with the output pointing at the separate left andright eye buffers. This applies to both types of stereo input.

There are a number of less common stereo video formats: even and oddpixels are for the left and right eye, respectively; even and odd scanlines are for the left and right eye, respectively, etc. Additions tothe video scaler sub-system can support these sorts of additional stereovideo formats.

III.G. EMD Awareness: Head Tracking

There are two fundamentally different sources of stereo imagery that maybe transmitted via stereo video formats. One is pre-computed orpre-photographed (film or digital, still or motion) left and right eyestereo images. With these, the stereo viewing matrices were bound at thetime of acquiring/rendering the images and cannot be easily changedafter the fact. Such stereo data generally cannot use real-time headtracking to produce different viewpoints into the stereo data.

The second type of stereo imagery is for example being computed inreal-time by 3D graphics rendering card(s), or acquired in real-time bya remote set of telepresence cameras: either several fixed cameras(multi-channel stereo) or two cameras with their motion slaved to themotion of the remote viewer 110 (e.g., a fast robot head). This secondtype of stereo can take advantage of accurate head tracking informationif it is available. Because in most configurations, an EMDS includes ahigh accuracy head tracker 120 portion, the solution reduces to aninterface problem: how to get the head tracking data from some part ofthe EMDS to the image generating system. Simple interface formats suchas USB™ are more than adequate to solve this problem but various formsof Ethernet, FireWire™, RS-232, RS-432, etc. also can work.

Let us further consider the case of a video source device that isrendering 3D graphics in real-time based on the continuously updatedhead tracking data provided by the EMDS 105. Such a system qualifiesunder most definitions as a “virtual reality system.” Later this casewill be split into different cases by the particular applicationinvolved.

If a video source device is capable of rendering stereo utilizinginformation provided by the head tracker component of the EMDS to createthe left and right eye view matrices, then it qualifies as a “headtracked stereo” display.

III.H. EMD Awareness: Augmented Reality

For applications that employ some form of digital still or video camera(this includes film cameras that have an augmented video camera, andstereo or multi-channel cameras), if a view from this camera isdisplayed to the user 110 via an EMDS, but with the user still also ableto see the real world in front of him, this describes the technologyreferred to as “Augmented Reality” (AR). The more physical and virtualcamera parameters are aligned, usually the better. Therefore, ensuringthat the video camera's field of view is matched to the field of view ofthe display on the EMDS is important. In one approach, the video camerais positioned just in front of the user's eye, blocking out all of thereal world, but the camera's output will re-display the view via theEMDS. Such systems also work if the physical video camera is positionedabove the user's eye, e.g., not blocking the normal physical view out ofthe user's eye. In another configuration, a video camera is worn on theuser's head and points down to a 45° half silvered mirror in front ofone of the user's eyes. Below the user's eye, there is a black material,so that the view not bounced off the half silvered mirror to the camerais essentially black. There are other ways to configure such a systembeyond those simple ones described here, e.g., use of still rather thanvideo cameras, stereo and multi-channel stereo cameras, other cameramounting points, other ways to achieve AR, etc. Augmented realityawareness was presented in this section because many differentapplications can use the results of AR capability in their systems.

III.I. Eye Mounted Display Levels of Awareness: Virtual Reality

If one takes most any EMDS 105, and puts black-out shades over the eyes,e.g., the eye(s) are only perceiving photons generated by the EMD, andthen uses real-time computer graphics rendering technology andtechniques, and the graphics image generator “camera view matrix” ismade to coincide with the head tracking data from the EMDS 105, thenthis describes the technology known as “Virtual Reality” (VR). The morethe virtual world parameters are aligned with the physical worldparameters (for example, how far down is the virtual floor depends onthe height and head movement of the user 110), the higher the realismand generally the better the results (and reduction of “simulatorsickness”).

Virtual reality awareness was presented in this section because manydifferent applications can use the results of VR capability in theirsystems.

III.J. EMD Awareness: Eye Tracking

Almost no systems in use today have the ability to take advantage ofreal-time extremely high resolution eye tracking data. Most systems thatuse eye tracking are specialty marketing advertisement evaluations, orfor visual science research. To the extent that a more general purposeapplication might make use of eye tracking data, it can use it for focusof attention, but even this has caused glitches in the past where theintent of the user 110 was not always reflected in their eyeorientation.

However, a graphics rendering system can take some advantage of knowingthe location and orientation of both eyes without explicit knowledge orinterface to the pseudo cone pixel data stream. Just knowing the generaldisplay resolution fall-off from the center of the eye for EMDS canallow a fixed density 3D rendering system still to obtain someperformance advantage through a number of techniques: tessellatingobjects less in areas of low resolution, applying lower cost shaders inareas of low resolution, applying higher cost shaders in areas of highresolution, applying lower sampling density if possible in areas of lowresolution, etc.

III.K EMD Awareness: Additional Object Tracking

Many other parts of the user's body and the physical world can betracked than just the user's head and eyes. When using an EMD forhead-tracked stereo display, it is convenient to have a 3D mouse. In thegeneral case, such a mouse would track in xyz the position of a “wandtip” or other point relative to the coordinate frame of the 3D mouse.The 3D mouse would also track the (three axis) orientation of the 3Dmouse body with respect to some coordinate system, usually the trackerframe physical coordinate frame. The 3D mouse would also have severalbuttons, etc. on it. For example, see Deering CACM “HoloSketch” for somedetails: Michael F. Deering. The HoloSketch VR Sketching System.Communications of the ACM 39(5), 54-61, 1996, which is incorporatedherein by reference.

A more general tracking of the user would be to use a “data glove”(e.g., Scientific American, October 1987, which is incorporated hereinby reference) where all the articulation of the user's finger joints aretracked, along with the xyz position and (three axis) orientation of theuser's hand(s).

Another general tracking would be to use a “body suit.” Now all of theuser's significant joints are tracked, which is the equivalent totracking all the xyz position and (3 axis) orientations of the majorlimbs and joints.

Tracking of objects other than the user's body can be performed, such astracking additional users. Tracking beyond this can be useful foraugmented reality, where the position and orientation of physicalobjects is made known to the controlling computer so as to allow theimage generator to properly occlude virtual objects when they go behindreal objects. One example application of this principle is “virtualsets” (The Virtual Studio: Technology and Techniques, Moshkovitz, Moshe,Focal Press, April 2000, incorporated herein by reference).

In many cases, the hardware and software already present within an EMDSsupporting head and eye tracking may be used to support tracking ofadditional objects, such as a 3D mouse, or at least expended on in acompatible way to support more complex tracking such as a body suit.

III.L. EMD Awareness: Pseudo Cone Pixel Data Stream

At the opposite end of the spectrum of systems that treat EMDS 105 as ifit were a simple flat LCD display, there are systems that are using thehead and eye tracking data to render 3D data directly to individualresolution varying pseudo cone pixels. Such systems could typically beadvanced virtual reality systems, or planetarium type displays. In bothcases, the external rendering hardware image generator intersects raysfrom the viewpoints to the surface of a sphere, as opposed to thesurface of a plane, as in most normal 3D graphics, or the surface of acylinder, as one possible built-in mode of an EMDS scaler. Such imagegenerators are not bound to use spherical rendering surfaces. Variouspolygonal (flat) piecewise approximations to a sphere will work as well.As one specific example, consider the case in which an image generatorcollection generates 48 channels of mono or stereo PCPDS onto a 48triangle spherical approximation. The 48 triangles are placed asfollows. First segment the sphere into eight equal parts by each of thethree coordinate planes, e.g. eight octants, e.g. +x+y+z, +x-y+z, etc.Now segment each octant by the three planes defined by x=y, x=z, andy=z. This will generate six triangular facets per octant (and six timeseight is 48). Now the image generators would use a standard flat imageplane (e.g., not spherical image plane) to generate 48 PCPDS.

IV. Product Classes Combining Electronics Devices and EMDSs

Now that various “EMDS awareness” options have been presented, thissection describes several classes of products that use EMDSs. Thefollowing is a rather long list, but it still is not exhaustive becauseEMDSs have the potential to replace nearly every existing category ofdisplay, as well as enabling many new ones.

IV.A. EMD Aware Digital and Film Still and Motion Cameras

Eye Mounted Display Aware: Stereo, Head Tracker, Eye-tracker, Wide Fieldof View, Augmented Reality, Pseudo Cone Data Stream

Many aspects of the consumer, prosumer, and professional categories ofstill and motion digital and film cameras, as well as motion digital andfilm television and movie motion cameras are shared. Most such camerashave a “viewfinder” of either a real world views (e.g., SLR cameras) oran image display (typically on small cameras, but also on SLR and otherprofessional digital cameras). In motion picture applications, there maybe more than one instance of the viewfinder. There may be an auxiliaryLCD panel displaying the digital image for the director, while thecinematographer looks through the camera's primary viewfinder (typicallyoptical).

To start with, a non traditional configuration of the camera and theuser 110 can improve the camera interface. For example, the camera mightbe mounted on the head of the user, with one of the user's eyes coveredby the camera and its lens, or covered by a tilted mirror reflectinglight up to a heavier camera mounted on the user's head. By having thecamera and the EMDS 105 display the camera foveal/peripheral pseudo conepixels to the EMD behind the occluded eye, in a non-zoom mode the EMDimage can look like a vignette image (by the camera's maximum field ofview). Because the other eye is not covered, the vignetting will mostlydisappear due to the stereo dominance of the uncovered eye. To show theuser what will be photographed, a small border could be rendered on theoccluded eye's EMD just outside the area of the camera's view. This willfurther indicate to the user where the camera will crop the scene whenthe user presses the shutter button and takes a picture or, in the caseof a motion camera, will show where the continuous images are beingshot. Various traditional displays of camera status (f-stop, speed,flash, etc) can possibly be displayed outside the active pixel area.

The previous example assumed a digital camera was present in the camerafor previewing or continuous shooting. However, it is becoming morecommon for film cameras (still and motion) to use an additional outputport (or the previous viewfinder port) to simultaneously shoot a digitalimage or motion sequence at the same time that film image(s) are beingrecorded. All of the description above applies to this case as well, andalso applies to digital as well as to film.

Zoom can also possibly be added to such a head-worn camera system. Forexample, the rectangular area being displayed on the blocked eye's EMDcan have its area shrunk to outline the correct narrow field of view ofthe zoom. Such an interface mode allows the user 110 to see normally,but the “capture” area will be smaller. This could be especially usefulin applications where the outside context is important, such as sportsor nature photography. Alternatively, the “insert” on the blocked eyecan be kept at its full field of view size, but the contents replacedwith the zoomed image. To reduce the effect of binocular rivalry, thenon occluded eye can be closed or blocked. Many wide field of viewlenses, so-called “fish-eye” lenses, have some amount of distortiontowards the edges of the frame. EMDs do not inherently suffer from suchdistortions, but these distortions can be added to the displayed imageto properly emulate what the fish-eye lenses lens is seeing or willproduce.

If the EMD is bright enough (and the variable darkness filter in theheadpiece can be darkened appropriately), it is possible to also displaythe rectangular outline of the camera's view, but in stereo. Usingstereo allows for another intuitive way of setting camera features. Thestereo depth of the rectangle can be made to be set at the current planeof focus. Thus, to focus the camera, one might adjust the focus ring,knob, in-out button, or other control, to move the rectanglerepresenting the current plane of focus to “surround” (with respect todepth) the objects one wants in focus.

If the shutter open time is too long, the user 110 could use a“chin-rest” on top of a mono/tri-pod to ensure stability during the longexposure.

Even if the camera is not mounted on the user's 110 head, e.g. hand-heldor on a tripod, a similar interface could be used with variousrestrictions (e.g., one or both eyes covered from the outside world.Alternatively, the camera's flat image might be presented as a rectanglefloating in space. This has advantages compared to present opticalviewfinders.

For video/motion picture applications, novel modes of “filming” may beenabled. Just as Steadicam allowed cameramen new, more flexible shootingopportunities, slaving a robotic arm held camera to an operator a shortdistance away could allow even more fluid and flexible shots, and withcamera/lens systems that are two heavy for an operator to wear. Camerasbeing flown by wire across a valley or other terrain might have natural,real-time control of the camera by a remote operator.

Such a still camera in use can be illustrated by using FIG. 12,reference 1200 again. The photographer 110 is wearing an EMDS 105, andhas set a physical coordinate frame on the ground near him. He hasadjusted the tripod 1250 to raise the EMDS aware camera 1210 to eyeheight, and uses the EMDS to see what the camera sees, rather than usingan optical or physical display viewfinder. Because the image displayedis eye tracked, the virtual viewfinder 730 could be made to appear atthe same high resolution as the camera. In this stand-up position, hecan change the zoom factor of the camera, change the focus of thecamera, pan or tilt it, and change other settings such as aperture andshutter time. After a (high resolution) picture has been taken, he canexamine the captured still frame, preferably at its full resolution,aiding the decision to keep the picture and/or take another shot.

Another possibility for novel camera control occurs when the camera alsotracks the motion of the user's eye(s). In one case, a still and/orvideo camera could be placed directly on the eye mounted display worn onor in the user's eye(s). Such a camera would automatically track themotions of the user's eye because it is effectively part of the user'seye(s). While such a eye mounted camera could be folded within the EMDusing some of the same optical folding techniques used in folding thedisplay optics of the EMD, such a camera would be necessarily limited inresolution and features compared to an external camera. Another way toobtain almost the same effect, but with full camera features would be tomount the camera to the user's headpiece, and then use motors to pan andtilt the camera to point in the same direction as the user's eyes, usingthe direction information from the eye tracking subsystem. Such a cameragreatly reduces the time and physical grabbing of an external camerawhen taking a picture; as an example a particularly gorgeous sunset canbe photographed with something as simple as a quick glance and a doubleeye blink.

IV.B. EMD Aware Stereo and Multi-Channel Stereo Still and Motion Cameras

EMD Aware: Stereo, Head Tracker, Eye-tracker, Wide Field of View, PseudoCone Data Stream, Augmented Reality

The interface described in the previous section could possibly bemounted to a stereo camera (still or motion), in which case the pre-viewimage could be displayed in proper stereo (at least proper for thecamera).

The interface preferably would include cameras covering each eye, butallowing enough situational awareness for a camera operator (consumer orprofessional) to walk around almost normally, snapping single or motionshows as desired. One example is shown in FIG. 15, reference 1500, wherethe photographer 110 wearing an EMDS 105 looks through a EMDS awarestereo camera 1510, with a virtual viewfinder 730 showing the scene 1520awaiting capture.

Multi-channel (e.g., more than two cameras) can be controlled via thesame user interface described above for two channel stereo. However,with a head tracker 120, intermediate interfaces that allow thephotographer to look through the plethora of stereo images being imagedat the same time could be useful. For example, in the still shot case,this could be used to make sure none of the cameras are seeing somethingthat they should not (e.g., a telephone pole, a close up leaf) that maynot show up on the other channels.

As described previously, the “virtual frame” or “virtual viewfinder”(730) can be used to delineate the edges of what is beingfilmed/digitized.

One advantage of multi-channel video cameras is that they can beespecially useful in remote situations where there is a significant lagbetween the array of cameras and the photographer. So long as all ncameras have transmitted a set of frames that can be examined inreal-time by the remote photographer, then by moving his head, thephotographer can get a good three-dimensional view of the remotelocation, even if the view is slightly out of date. This can be superiorto telepresence robots in which the communications time lag is apparentto the photographer, e.g., move one's head, wait for images to catch upto the new head position, which can result in nausea and/or a poorunderstanding of the remote scene.

The above applies to most, if not all, approaches to “arraying” theremote cameras: n cameras in a linear line, n cameras along asub-circle, n by m cameras in a two dimensional array, n by m by zcameras in a three dimensional array, etc.

IV.C. EMD Aware Cell Phones and PDAs.

EMD Aware: Modifying the EMDS Scale, Elimination of Half ofHead-Tracker, Stereo, Pseudo Cone Data Stream

While current cell phones and many PDA's allow one to access the webwirelessly from a small device, it is less web-surfing thanweb-pogo-sticking. This is due to the very tiny (usually LCD) displayspresent on these devices, which are tiny both to fit in the availablespace of a tiny device, and to keep cost and power consumption down.

However, if someone is wearing an EMDS 105, at a relatively small costeven a cell phone can have scaler sub-system circuitry added to sendpseudo cone pixels to the EMDS, which virtual image will visually appearin size and functionality more like a full computer web-browser, oremail application, spreadsheets, databases, maps and directions,still/video camera viewfinder, etc. As described earlier, cell phones,PDAs or other devices do not necessarily have to support the externalhead-tracking components, further simplifying the construction. Theactual circuitry to make a cell-phone EMDS aware has mostly beendescribed in the section titled “Modifying the EMDS Scaler Hardware”above. It is then up to only software layers to support a web browserwith various features. As more and more (high end) cell phones (forexample, the Apple iPhone™) have their runtime systems based on generalpurpose computer operating systems, this need not be an onerous task.

FIG. 16 reference 1600 shows a cell phone user 110 in public wearing anEMDS 105, holding an EMDS aware cell phone 1610. In this example, theactual virtual image that the cell phone causes the EMDS to display is asingle two dimensional image in stereo (e.g., the same image in botheyes, but offset), apparently about six feet (or whatever distance isprogrammed) away, as seen in reference 1600. The number of pixels in theimage preferably is at least minimum PC size (e.g., VGA, 800 by 600) orlarger, and with the field of view of each pixel as large or larger thana pixel viewed on a PC display when viewed at normal reading distances(e.g., 80 dpi at 20 inches).

FIG. 17 reference 1700 shows a cell phone user 110 in public wearing anEMDS 105, holding an EMDS aware cell phone 1610 sitting at a bus stopbrowsing the web and checking email.

IV.D. EMD Aware Heads Up Display

EMD Aware: Resolution, Wide Field of View, Stereo, Head-tracker, AugmentReality

A “heads up” display is generally a display superimposed on the exteriorview, typically out of a vehicle, typically also displaying variousvehicle instruments. The display usually is at optical infinity, so thata vehicle operator does not have to take the time to change frominfinity focus to short distance focus on an interior to the vehicledisplay, and then refocus back out again to infinity. Historically,heads-up displays were deployed in expensive vehicles, e.g. fighterjets, but now can be found in cars. Conceptually, a heads-up display issimpler than a EMDS 105, as no eye or head tracking is required, and theeffect is produced by projecting an image of the instruments of interestonto the interior of the window in front of the vehicle operator,corrected for infinity.

However, a heads-up display is just another form of display, and couldbe emulated by an EMDS 105. Thus as EMDS become appropriate for variouskinds of vehicles and their operators, the advantages of EMDSs couldcause replacement of heads-up displays with EMDSs. Specifically,heads-up displays are a limited form of augmented reality. Having anEMDS instead could allow more sophisticated type of data to bepresented, possibly in all directions. FIG. 18 reference 1800 shows apedestrian 110 using an EMDS 105 and an EMDS aware cell phone 1610 todisplay in real-time directions from a web based mapping site. But alsoconsider navigation systems when driving an automobile.

FIG. 19, reference 1900, shows an augmented reality automotivenavigation. Here the real world 1910 has a virtual overlay of textinstructions 1920 and graphics 1930 showing the path ahead. If the EMDSis operating in stereo, then the graphics data, such as the curved arrowshown, will be correctly mapped in stereo to the apparent surface of theroad. Of course, as with existing navigation systems, this may befurther augmented by audio voice instructions. FIG. 19 only concerns thevisual portion.

IV.E. EMD Aware Video Kiosks and Digital Signage

EMD Aware: Modifying the EMDS Scaler HW, Resolution, Wide Field of View,Stereo, Head-tracker, Augmented Reality, Virtual Reality, Eye-Tracker,Pseudo Cone Pixel Data Stream

Currently, more and more fixed signage used for advertising in shoppingcenters and stores is being replaced with digital signage. In yetanother application, as a person with an EMDS 105 passes by stores,three dimensional images of wears can be displayed by free spaceconnecting to an EMDS—in effect, a virtual store window.

This is shown in FIG. 20, reference 2000, where a shopper 110 wearing awireless EMDS 105 comes into range of the next store in a mall. Then athree dimensional view of the virtual store window 2010 inside thevirtual display 730 is transmitted to the shopper. In this example, thestore window contains a local tracker frame 230 and wireless pseudo conepixel data stream transceiver 228. Some sort of standard interface tothe EMDS is assumed so that the pseudo cone pixel data streamtransceiver 228 can interface to the EMDS 105.

IV.F. EMD Aware Laptop and Palm-top Computer

EMD Aware: Resolution

Many laptop computers have relatively large displays, but this comes atthe cost of price, power, and weight. Also, the size of the pixels donot scale with the increase in pixel count, so the pixel density ofrelatively high resolution laptops is frequently wasted, unless attachedto a larger external monitor. An EMDS 105 can emulate such a monitor,but in a portable and low power package. In the simplest case (forcompatibility of the installed base), the EMDS's video input is pluggedinto the video output of the laptop; and a portable collapsible trackerframe 230 can be placed on (say) the espresso table next to the laptop.This situation is shown in FIG. 21, reference 2100, where a personwearing an EMDS 105 is viewing the video output of their laptop computer2110 in a (relatively) large virtual display 730

The power for the electronics for the EMDS 105 could be externalrechargeable batteries or powered by the laptop over USB, for example.This is not as onerous a requirement as it sounds; as when using theEMDS instead of the laptop's integral LCD display, the backlight and LCDdriving of the integral display can be powered off, making the excesspower available for the EMDS. While the best display is obtained byhaving the laptop output video at its highest native resolution, when alower resolution is sufficient to the current task, incrementaladditional power can be saved by having the laptop output, and the EMDSprocess, a lower resolution image.

In an alternate approach, a laptop need not have a traditional integraldisplay but could assume instead that an EMDS 105 will be used. Such adevice could also have the tracker reference frame 230 built in, forexample only requiring the Y axis portion of the frame to be extended inuse. Such a laptop might also have an optional detachable traditionalLCD screen available.

So called “palm tops” are full function computers, but with a physicalsize not much more than four or so times larger than a PDA. Such deviceshave tiny low resolution screens (though much better than cell phonesand PDAs) and tiny keyboards. Dispensing with the tiny display andreplacing it with an EMDS, a palm-top could have a much closer to fullsize keyboard fold out from the device, and have a relatively largeimage display thanks to the EMDS 105. There are also “virtual keyboards”that use light to create a keyboard on any surface that the palm topprojects to.

In addition, most laptops are plugged into the wall 95%+ of the time,even if it is an airline lounge, or on the newer airplanes with ACsockets at each seat. So long as an AC outlet is available, there is noproblem powering the EMDS 105 with the larger and more power hungryscaler “black box”, without affecting the laptop power.

There is another intrinsic advantage of EMDS 105 mounted to laptops:security. While some business travelers read or edit internal companydocuments or spreadsheets, in many cases this risks a breach of companyconfidentially, as other passengers can see the display too. This is nota problem for an EMDS since other passengers cannot see the virtualdisplay of an EMDS.

While most current laptop use is mono, specialized stereo laptops doexist. When a two eye EMDS 105 is used with a laptop, the laptop can runapplications in head tracked (or not) stereo display mode.

IV.G. EMD Aware Wearable Computer

EMD Aware: Modifying the EMDS Scaler HW, Elimination of Half ofHead-Tracker, Stereo, Pseudo Cone Data Stream

The biggest limit on wearable computers has been the limitations ofpreviously wearable display technology. Weight is not an issue. Mostpeople are wearing considerably more additional pounds of fat than theweight of a wearable computer configured, say, as a thick belt. EMDS 105could be what wearable computer devices need as an enabling technologyto make them a realistic alternative to more traditional fixed locationcomputers. However, just as is the case for many other applications, forextra power the wearable computer does not have to know about the EMDSto still garner many of its advantages, but an overall better systemdesign results when the EMDS is designed in from the ground up. Theoverall system can be similar to an EMDS aware laptop; indeed justplacing such an aware laptop into a small backpack while wearing an EMDSis a sort of wearable computer, other than keyboard and pointing inputfunction devices. Wearable alternatives to these exist (e.g., cordkeyboards in pockets). It is making the large relatively high resolutiondisplay available that is most of what this market currently lacks.

An interesting alternative to a wearable computer is an office or labenvironment in which many computers are connected together viaconventional networks, but their video outputs are available to beplaced out on a short range spread spectrum (or equitant free-space highdata rate technology) transmission that couples to the worn EMDS 105.Keyboards and mice can still be used via the lower bandwidth backchannel. In this case the computer is essentially “in the walls”, andeven which computer one connects to does not matter so long as it canaccess the user's 110 desired data (email, web, etc.).

IV.H. EMD Aware HDTV Display

EMD Aware: Resolution, Wide Field of View

While HDTV systems are coming down in price so that a much larger numberof consumers can afford them, the low cost low end still have to makemany quality compromises in video quality, video brightness, colorfidelity, bulb life, and other features greatly desired by the homemarketplace. Also, just the amount of electrical power and cost ofreplacement projection bulbs required for operation of these devicesover their lifetime can exceed the initial cost of the device. Consumersdesire 1920 by 1080 pixel resolution, with a refresh rate of at least 60Hz. To avoid having to internally perform 3/2 pull-down (with itsassociated negative artifacts) on display of motion pictures originallyshot at 24 frames per second, many displays are moving to 120 frames persecond internal display rate, allowing each original 1/24^(th) of asecond frame to be displayed an integral five times before the nextframe is displayed. 72 Hz or 96 Hz display rates are other effectivealternatives. The advantage of 120 Hz display is that the existing 60 Hzframe transport video formats can be used to move the frame from theplayback device (HDTV sources) to the display.

Given that the average number of people sitting in front of an HDTVdisplay at any given viewing time is approximately 1.1, a high qualityEMDS 105 is a viable alternative for both higher quality display, andcomparable price. It is also conceivable that, when two people arewatching a video together, each sporting an EMDS of their own, thevirtual image in space can be made to coincide so that the two viewers110 can point to something in the display in sync in physical space witheach other.

There are many advantages to EMDS 105 HDTV viewing. No large heavyfolded rear-projector boxes need be set-up, nor front projectors on theceiling along with drop down screens. One just jacks in (or connects toa free space transmission point) of the home video network, and sitsback and watches a movie. If no one else is using the room, the lightscan be dimmed. Otherwise the viewer 110 can darken the see-through inputin any of the ways previously discussed. Better sound will still comefrom several external speakers (which anyone else in the room would haveto put up with), but because the viewer is head-tracked, multi-channelaudio headsets or ear inserts can provide a high quality threedimensional sonic experience, without bothering others in the room.

FIG. 22, reference 2200, shows a tracker frame 2260, left 2205 and right2210 stereo speakers, and a rack full of HDTV and EMDS equipment. Thisis an example set-up suitable for most of the home entertainmentexamples to be described as follows. Although only two speakers areshown in this specific example, this is a stand-in for most any otheraudio environment, from headphones through 5.1 sound and 7.1 sound. 7.1sound means speakers located at: the left, the center, the right, theleft side, the right side, the left rear, the right rear, and one ormore sub-woofers (the “0.1”). The components in this example rackinclude the following: a satellite dish HDTV TV receiver 2205, a cableTV HDTV receiver 2220, an over the air HDTV TV receiver 2225, an audioamplifier 2230 for the speakers, a personal video recorder (PVR) 2235(could instead be built into one of the HDTV receivers), a pseudo conepixel data stream transceiver 2240 (or a hardwired connection), atracker frame 2245, an EMDS scaler unit 2250, a BluRay™ (or HDDVD™) (orjust DVD) disc player 2255, a personal computer 2265 with an embeddedimage generator.

FIG. 23 reference 2300 shows an at-home HDTV set-up, with the virtualdisplay 730 aligned with the wall. A physical coordinate tracker frame230 is shown as well. The viewer 110 is wearing an EMDS 105, which isfree-space transmitter 228 attached to a HDTV output device 2210. Alarge number of such output devices are already in existence: HDTV overthe air broadcasts, and/or satellite or cable HDTV, and/or HDTV PVR,HDTV playback device (Blu-ray™ and/or HDDVD™), or HDTV switching devicesconnected to some or all of the above. Computers and game consoles canalso be plugged in here as well, but detailed advantages of suchconnections will be presented in later sections. In the case of FIG. 23,the EMDS aware HDTV output device 2210 is a representative for allpossible such devices. Note that none of the non-game HDTV devices needto be EMDS aware.

IV.I. EMD Aware Day of Release Motion Picture Display

EMD Aware: Resolution, Wide Field of View

The movie industry's current business models are shifting. There is asignificant segment that will wait for the DVD (or HDDVD™/Blu-ray™) diskto come out, or to show in HBO™ or SHOWTIME™. Some movies are havingtheir release on DVD occur the same day the movie starts in theaters.Just as many people have a better sound system at home that the Cineplexdoes, high end home HDTV or EMDS 105 can potentially provide higherquality images than those available at the Cineplex, especially afterthe film has been run through the projector several times. Many theatersare moving to digital displays, replacing film projectors, but consumerdisplay technologies are out-stripping the more constrictive theaterdisplay.

As a concrete example, an EMDS has the potential to produce a higherquality image than IMAX™ displays, let alone a “mere” 35 mm or 70 mmprint, or a 2048×1080 digital projector, which is the current maincommercial theater digital projector standard. In one business model, onthe day of release, encrypted versions of the movie are sent tohouseholds that paid for the privilege. They can see the movie openingnight from the comfort of their homes, and since the video data can beencrypted as far as all the way to transmission to the EMD, the moviecompanies will have lessened piracy worries. If there is EMDS awarecomponents in the provider electronics, one can charge not just a singlepay-for-view price, but a pay-for-view price multiplied by the number ofEMDS viewers 110 present.

IV.J. EMD Aware 3D HDTV Display

EMD Aware: Resolution, Wide Field of View, Stereo

The motion picture industry is also placing some emphasis on modernstereo camera shooting and theater displays. However, most EMDSs 105(when used with two EMDs) are inherently stereo and can be higherresolution than film to boot. If 3D versions of films are released in aconsumer format, the home EMDS can be used to display them. FIG. 24 isthe same as the home HDTV theater of FIG. 23, except that different HDTVimages are being presented to each eye of the viewer 110. The 2D virtualdisplay 730 of FIG. 23 is replaced with a 3D virtual (stereo) display730.

IV.K. EMD Aware Large Screen Format Display, and 3D Display

EMD Aware: Resolution, Wide Field of View, (Stereo)

As described above, an EMDS 105 can have greater field of view and pixelresolution than 15 perf 70 mm film, which is what is used to produceIMAX™ films. Many IMAX™ films are in 3D, but once again this is anatural format for EMDS. Commercial same day of release distribution ofmovies in IMAX™ or IMAX™ stereo format is another way to keep theatricaldistribution revenues up. Just as in the more traditional motion picturecase, an EMDS has the potential to produce superior displays than thetraditional film or new (constant resolution) digital cinema projectors.Once again, a direct to consumer marketing model may become a viabledistribution model for the movie business.

FIG. 25 reference 2500 shows an at-home HDTV set-up, with the virtualdisplay 730 aligned with the space in front of the viewer 110. Aphysical coordinates tracker frame 230 is shown as well. The viewer 110is wearing an EMDS 105, which is free-space transmitter 228 attached toa large screen format stereo supporting output device 2510. Note thatall this is similar to the home HDTV theater example of FIG. 23. Themain difference is that the virtual display 730 covers more of theviewer's 110 field of view, and the content playback device 2510 has ahigher resolution and assumed field of view than standard HDTV.

No matter how high the resolution of the input image, the daisy-chainedpseudo cone pixel data stream has the same data rate (in oneimplementation). The typical problem with transferring much higher thannormal input images has been addressed in a number of ways. The mostcommon is to replicate the existing highest resolution interface. Thistype of high resolution input is supported in the scaler sub-systemdescribed above for a typical complete EMDS. Each scaler sub-sub systemis happy to accept one n^(th) of the input data, and output thecorrectly processed portion. This feature may be used to support displayof “4K” and “8K” video formats and above. (The 4 and 8 refer to thewidth of the video frame format in pixels; the height of the framedepends on the aspect ratio of the format.)

IV.L. EMD Aware Sports Display

EMD Aware: Resolution, Wide Field of View

At sports stadiums, watching the game live can be inferior to watchingit televised, because video cameras can get much closer to the actionand can show instant re-plays. This is partially addressed in manysports stadiums by the presence of one or more very large, very brightdisplays (typically super-bright LED displays in newer installations),so that at least replays can be shown, as well as other officialfunctions. As shown in FIG. 26 reference 2600, if a sports fan 2610 iswearing an EMDS 105, an in-stadium set of HDTV+ free-space stereo videochannels can allow the attendee to switch from a live stereo view of theaction from their seat (e.g., naked eyeballs), to zoomed in displays andstereo high-resolution replays 2620, as shown in the virtual display730. Not shown in this figure are local physical coordinate frames 230and free-space transmitters (or plug in jacks). Such displays can alsobe available at distant location pay per view sites.

IV.M. EMD Aware Immersive Virtual Reality Display EMD Aware: Resolution,Wide Field of View, Stereo, Head-tracker, Virtual Reality

EMD non-aware virtual reality applications can make use of an EMDS 105by rendering and displaying fixed (vs. variable) resolution images foreach eye, with the view transformation matrices for rendering derived inpart from the real-time head tracking offered by most EMDSs. Used thus,to the non-aware application, an EMDS looks like a high resolution formof a head mounted display (HMD), with integral head tracking. Thus EMDScan support “legacy” HMD applications and image generating devices.

To put some numbers on the human eyes fields of view are for help inunderstanding some of the following paragraphs, FIG. 27, FIG. 28, andFIG. 29 are included. FIG. 27, reference 2700, is a polar plot showinghorizontal and vertical limits in degrees of what the left eye can see.The solid line 2710 is the limit of the vision of the left eye. The lefteye's blind spot is 2720. The dashed line is the limit of the right eyefor comparison. FIG. 28, reference 2800, is the same but for the righteye. The solid line 2810 delimits what the right eye can see and theright eye's blind spot is 2820. The dashed line is the limit of the lefteye for comparison. In FIG. 29, reference 2900, the solid line 2910shows the area of stereo overlap, i.e., the portion of visual spacevisible to both the left and right eyes. Note that viable displays donot need to cover these visual areas entirely. Many eye glasses andcontact lenses artificially narrow the field of view available withoutnotice by the human 110.

However, beyond legacy HMD emulation, it is also conceivable toconstruct 3D graphics rendering chip to take advantage of the variableresolution pseudo cone pixel array that is all that a particular eyeneeds for “full image resolution” for a given frame. The modificationsare described in U.S. Pat. No. 6,525,723, “Graphics system which renderssamples into a sample buffer and generates pixels in response to storedsamples at different rates,” which is incorporated herein by reference,but the result can be a reduced rendering load along with a very widefield of view.

“Visual immersion” is defined to start at a bare minimum of 65° field ofview, with 85° being better. In theory, an EMDS 105 can present the samefield of view that the real world does, e.g. limited nasally by the edgeof the noise, and temporally by the temporal edge of the eye socket.This can be as high as 165° per eye, and 190° or more for both eyes.Practically, supporting a field of view out to the inner edge of thesunglasses like portion of the headpiece is sufficient, if present. Themaximum vertical field of view is approximately 50° vertically fromlevel both up and down, e.g. 100° full field vertical field of view.Sunglasses typically afford approximately 100° horizontal field of viewper eye, and considerably less vertically.

However, many types of eyewear that have somewhat limited through view,still leave a lot of peripheral vision “outside the frame”. EMDs canwork the same way. Most designs do not artificially block vision fromangles larger than the display can generate. But while this is OK forHUD and some augmented reality displays, it is not OK for immersive andmany augmented reality displays. Effectively having some portions of“reality” vanish at an angle at which the real world is still visible isnot good. For these applications, the simplest way to prevent this is toclose off all physical world view angles anywhere where the EMDS cannotdisplay an image also. Effectively this means dark sides (and tops andbottoms) to the portion of the headpiece worn over the eyes. Similarexamples can be found in welding goggles and more extreme sunglasses,all for the same reason. Only the portions of the visible world seenthrough the main lens can be allowed into the eyes. So long as theoverall field of view is fairly wide (e.g., horizontally 85° to 100°)then the user 110 is not likely to even notice the narrowing of thefield of view. One reason for this is that the eyeball can rotate in itssocket only so much. The field of view that one can point one's fovea atis well less than 165°. In fact it is not much more than 100°. The restof the view is low resolution peripheral vision. The portion of thehorizontal field of view that can be perceived in stereo, e.g. areasthat both eyes can foveate on, is even less, approximately 60°. Asanother example, most prescription eyewear does not correct for visionmuch wider than 100° for this very reason. Another important point isthat the “shutter glasses” stereo eyewear that is used in nearly allimmersive projective display environments (CAVETMs, Virtual Portals,etc.) have only a 100° horizontal field of view, and yet very strongimmersive effects are induced. The far periphery vision of the human eyeis actually one of the easier for an EMD to display to (very lowresolution, short projection throw distance or equitant). However, thereality is that if the EMD is wide enough to not induce “tunnel vision,”then the user will not feel that anything is missing. Of course, thereare situations, such as certain simulator training, where evenperipheral vision is important to the task, and EMDs that cover the full160° width of the field of vision are mandated

The “presence” or realism of current technology virtual reality displayshas had as a primary (but not only) limit the resolution and field ofview of the display systems. EMDS 105 could surmount this main obstacle,and also provide higher quality head tracking data than low cost VRsystems. While VR is not the main initial market for EMDS, low costwidely available EMDS could greatly affect the VR marketplace, possiblyenabling a number of new or previously unattainable applications, aswell as greatly improving the effectiveness of the few existing markets.

Another example of full immersion virtual reality is shown in FIG. 30reference 3000. Here a viewer 110 is completely immersed on all sidesand directions of view by a spherical stereo virtual display 730. Theviewer would usually be wearing black shades so that only the virtualworld enters his/her field of view. In fact, unless it was also modeledin virtual space, the viewer's 110 body 3020 would not be visible to theviewer 110. The 3D graphics imagery is generated by a graphics renderingdevice 3010 (the game console).

However in order for a system to render at maximum foveal resolutionacross the entire field of view offered by EMDS, this would meanrendering as many as half a billion pixels per frame. Thus to takeadvantage of the wide field of view offered by EMDS 105, as describedbefore, the rendering device 3010 preferably is pseudo cone pixel aware;e.g. capable of variable pixel density rendering, directly renderingpseudo cone pixels. Such a system would not need the standard black boxscaler device. That computation could be built into the graphicsrendering device.

IV.N. EMD Aware Augmented Reality Display

EMD Aware: Resolution, Wide Field of View, Stereo, Head-tracker,Augmented Reality

Many instances of EMDS 105 are inherently see-through. This allows themto function as augment reality (AR) displays. For some augmented realityapplications, only a small amount of low resolution graphics is neededand could possibly be provided by current off-the-shelf renderingsystems. However, for other applications, the augmented reality displaymay have to be five times or more brighter than the physical environmentas viewed from the inside of the headpiece. The reason for this is thatat intensities less than this multiple, the real-world corrupts orbleeds through the virtual display. Colors in particular are easilycorrupted by different colors in the environment. Thus if the color ofvirtual objects is important in an augmented reality task, then thebrightness threshold is important.

An example of augmented reality can be found in FIG. 32 and FIG. 33.FIG. 32 is a street scene with a soldier 110 looking around. FIG. 33 isthe same street scene, but with objects that were hidden drawn onto theEMD as augmented reality. The drawn objects include an aircraft 3310that was hidden by the clouds and a tank 3320 that was hidden by abuilding.

There are several other ways to merge the real and virtual world. Asdescribed in Michael F. Deering. High Resolution Virtual Reality. Proc.SIGGRAPH '92, pages 195-202, 1992 and U.S. Pat. No. 5,446,834 “Methodand apparatus for high resolution virtual reality systems using headtracked display,” which are both incorporated herein by reference, if itis possible to allow through a varying amount of the real world light ona pseudo cone pixel by pseudo cone pixel basis, including fullblack-out, then the virtual display need only be as bright as thesee-through environment. Another method is to use high resolution videocameras mounted just in front of the user's 110 eyes, to capturereal-time video images of the real world. Then the virtual world and thephysical world can be mixed in video space, and the results sent to theEMDS 105. Finally, with accurate enough cameras and other sensors, it isconceivable for the system to reconstruct the local physical world as acomputer graphics data base. Now the virtual world database can bemerged with this, and then both can be rendered together. This can leadto better integration of the physical and virtual worlds. Shadows ofreal-world objects fall correctly on virtual objects, and visa-versa.Physical transparent objects can layer in front of virtual objectscorrectly.

IV.O. EMD Aware Video Game Software Running on an EMD Non-Aware VideoGame Platform

EMD Aware: Stereo, Head Tracking, Wide Field of View

For the purposes of this section, we will refer to all EMD non-awaregame platforms as just “the game platform.” Thus this term refers to allpresent and many future PC gaming platforms, home console gamingplatforms, hand-held portable gaming platforms, portable gamingplatforms, and any other device (including cell-phones) with some formof standard or non-standard video output, where video games have or willbe able to be played, but without EMD awareness.

So long as there is reasonable speed input to the gaming platform (USB,or special formats, etc.) then accurate real-time head position can beobtained by new “head tracked aware” software on old non-aware devices.The existing video output of the game platform would be plugged into oneof the video inputs of the EMDS 105, and the head-tracking data outputstream would be connected to an appropriate pre-existing data input porton the gaming platform (possibly with a format changer/decrypter blackbox). Now the new video game software can take rendered frames of thevideo game with computer graphics viewing transforms that take intoaccount orientation and some position information, as well as the widefield of view. While this will work well for games in which most objectsare relatively far away, it may be possible for the new software to letthe EMDS 105 know when left vs. right eye video frames are being output,making the game stereo (even if the gaming platform did not alreadysupport stereo display). It is also possible to have a game that is now(or was already) in stereo, but not head-tracked.

An example of an EMDS non-aware video game being played on a hand-heldEMDS aware game device 3110 is shown in FIG. 31 reference 3100. In thiscase, is it assumed that the EMDS 105 includes a battery powered scaler(not shown). The video out from the game device 3110 in the low end iswired in this example, with this physical connection 3120 including aquick disconnect cabling interface (e.g., two magnets) that passesthrough this scaler and to the rest of the EMDS. The hand held gamedevice 3110 is assumed to have been retrofitted with a physicalreference frame device. The game player 110 seated on the park bench3130 plays existing and new games on a visually wide angle virtualdisplay 730. In this situation the head tracking could be used tostabilize the virtual display 730; but not to produce any head-trackedstereo display effects on the gaming device 3110.

IV.P. EMD Aware Hardware and Software Video Games on Various Platforms:Hand Held Portable, Portable, Console, Deskside PC

EMD Aware: Modifying the EMDS Scaler HW, Resolution, Wide Field of View,Stereo, Head-tracker, Virtual Reality, Eye-Tracker, Pseudo Cone PixelData Stream

If a gaming platform is aware of all of the aspects of an EMDS 105, itcan utilize a variable resolution 3D rendering device, and directlygenerate pseudo cone pixels on a spherical or nearly sphericalbackground. To a first approximation there is no difference between sucha video game platform and virtual reality display. The differences are,as usual, game applications can cheat in ways not always possible withmore general purpose display systems. The “simplifications” that agaming platform might make include: always completely blocking out lightfrom the physical world, rendering 3D graphics at less than the fullavailable (variable) resolution, pre-computing graphics simplifications(e.g., pre-lit radiosity), not rendering distant parts of theenvironment if the render load gets too high, using custom shaders tofake more complex lighting and shading effects, etc.

Also to a first approximation, deskside PCs and non hand-held gamingconsoles can be considered the same once a game is running. In bothcases, to take advantage of EMDSs 105 the 3D graphics rendererpreferably directly renders pseudo cone pixels, with a low data ratelink into the system to accept and process the head and eye trackingdata. The game software preferably takes advantage of the displaycapabilities, and keeps a high enough reality factor to minimize oreliminate “simulator sickness.”

Portable gaming platforms are usually just luggable versions of consolegame systems, and as such can utilize the EMDS 105 in the same manor.However, some are battery powered, and then the power consumption of theEMDS can be a factor. In the lowest battery power environment, such ashand-held game devices, the EMDS might have a simplified and low powerscaler sub-system; not unlike the sub-system that goes into cell phones.In a higher battery power environment, the scaler module might becompletely eliminated, so long as the 3D graphics component is capableof directly generating pseudo cone pixel streams.

One example is to re-consider FIG. 31 in which the hand held batterypowered game device 3110 is now EMDS aware. It would still display videoimages the EMDS 105. But now, also there would be a physical worldtracker frame 230 built into the game device 3110. The virtual display730 is a frame floating in front of the player 110. The game contentwould now be full head tracked stereo.

IV.Q. EMD Aware Simulation Systems: Flight, Tank, Dismounted Infantry,Homeland Defense, Firefighting, etc.

EMD Aware: Modifying the EMDS Scaler HW, Resolution, Wide Field of View,Stereo, Head-tracker, Augmented Reality, Virtual Reality, Eye-Tracker,Pseudo Cone Pixel Data Stream

Historically, military simulators have traditionally used multiple videoprojectors and rendering units, to provide the resolution and 3Dgraphics performance required to meet their needs. HMDs have usually nothad the resolution or field of view necessary for the simulation tasksat hand.

However, EMDSs 105 have the potential to alter this. Mounted with 3Drendering systems with variable resolution rendering capacities so thatthey can directly render to the pseudo cone pixels, EMDSs could providebetter resolution and field of view than existing simulators.

If, for example, a physically real cockpit is present in a simulator,with computer generated imagery visible only outside the windows in thecockpit, then the EMDS 105 could match a display image in space from agiven image generator to a particular window. Other simulators may onlyhave the controls built in the physical world, and place all the rest ofthe vehicle being simulated in the virtual world.

FIG. 32 reference 3200 shows an individual dismounted infantry soldiertrainee 110 wearing a wireless EMDS 105 in a completely immersivevirtual environment. In this case, the virtual environment 730 includesa street 3220, buildings 3230, trees 3250, and clouds 3240 in the air.In this environment, many different training tasks could be simulated,possibly including linking in of multiple other dismounted infantrysoldiers, tank simulators, aircraft simulators, CCC (Command, Control,and Communications), etc.

FIG. 33 reference 3300 is similar to FIG. 32, except in this trainingscenario, simulation of augmented reality display of tactical data isincluded: an aircraft 3310 that was hidden by the clouds, and a tank3320 that was hidden by a building. Other possible uses of augmentedreality are less direct. Three dimensional terrain maps can float inspace in front of the soldier 110 with various points of interestmarked. Location blips of the other soldiers in his/her platoon can bemarked, as well as any civilians or enemy soldiers sighted. The EMDS 105does not find the hidden enemy. It just allows the display of thistactical information in a readably useable way.

FIG. 34, reference 3400, is an example of a virtual Command, Control,and Communications (CCC) center. The displays on the walls are notphysical, but virtual screens 3010. The three dimensional topographicmap 3020 hovering above the table is also a purely virtual object. Onlypersonnel 3030 wearing EMDS 105 see the data 3020 being presented.

Training systems for firemen and police could use a similar physicalset-up as described above.

IV.R. EMD Aware Real World Systems: Flight, Tank, Dismounted Infantry,Homeland Defense, Firefighting, etc.

EMD Aware: Modifying the EMDS Scaler HW, Resolution, Wide Field of View,Stereo, Head-tracker, Augmented Reality, Virtual Reality, Eye-Tracker,Pseudo Cone Pixel Data Stream

Because of their inherent small size and power requirements, EMDS 105should be of interest in a variety of military tasks in the field. EMDScould allow large, highly detailed maps to be brought up for display,with active icons representing pertinent objects and areas. In someapplications, the best map may be a three dimensional augmented realitymap, showing objects directly. While high cost fighter jets have hadcomplex heads-up displays for years, such complex see-through displayshave been too expensive or cumbersome for use with lower cost vehicles,let alone individual dismounted soldiers. EMDSs have the potential toalter this, and along with the ever shrinking cost, weight, and powerrequirements of portable computational and communication elements, muchhigher functionality displays might be deployed at all levels ofmilitary tasking. This also applies to expensive, heavy “situation”rooms. With an EMDS per officer, as much situation can be displayed in alinked system as is desired. This allows such situation rooms to bedeployed much more quickly and closer to the front.

As an example, FIG. 32 can now be reinterpreted as a soldier 110 in areal street in a real town; and FIG. 33 can be reinterpreted as showingaugmented reality tactical overlays showing the hidden presence of areal aircraft and a real tank. Firefighting and police action could alsohave dynamic overlays of things that they cannot see directly, becauseof smoke, or because the police are hunkered down in a safe position.Remote cameras can show them what is happening in the danger area.

IV.S. EMD Aware Real World Systems: Command, Control, and Communications(CCC) center.

Military and civilian Command, Control, and Communications (CCC)applications traditionally have been large rooms with multiple largedisplays covering most of one large wall. Currently, these displaystypically are short folded depth rear screen video projectors. EMDS havethe potential to emulate almost any display environment, including thisone.

Each wall display is displaying the results of an image generator videofeed. With an EMDS, each would feed into a scaler (e.g., per viewer),and this display could be re-arranged in apparent real space position,size, and orientation as desired. Such a “virtual CCC” has the advantageof being very quick to set up, allowing military applications to moveCCC physical locations closer to the action. FIG. 34, reference 3400shows a hybrid display. All the wall displays 730 are virtual displays,but half of the room is “virtual”, representing another distant CCCcenter. On the tabletop shared between the two rooms, a 3Dtopological/map 730 display is shown.

IV.T. EMD Aware Full Scale Industrial Design Display

EMD Aware: Resolution, Wide Field of View, Stereo, Head-Tracker

In automotive body design, size matters. A half scale clay replica of apotential automobile body will not allow proper decisions to be made.The prototype must be visually full scale. Presently this isaccomplished in a large dedicated room with multiple panels of rearscreen video projectors, many-times in head tracked stereo. If stereodisplay with active head tracking is used, the display system becomes avery expensive single user system. A larger audience can view stereoobjects in the room with active or passive glasses, but with incorrectstereo viewpoints. With EMDS 105 technology, any designer, engineer,marketer, or executive can potentially use their personal EMDS to reviewlife size designs together or alone whenever they like.

An example of such a system is shown in FIG. 35, reference 3500. Herethe virtual automobile 3510 appears on a rotating (or not) circularpedestal 3520 to the engineer 110 and the executive 110 through theirEMDS 105 and private pseudo cone pixel data stream 225 corresponding totheir unique position and orientation with respect to the virtual car.The same would hold if the room was filled with additional viewers.

The previous example used automotive body design as just one example ofwhere EMDSs 105 can be of use. Similar scenarios can be applied to anydesign, from bicycles to jumbo jets, from ski-boots to kitchenappliances, and include not just the exterior look, but the design ofthe internal working parts.

One can also design objects at full scale from the inside. FIG. 36 showsa designer 110 designing the virtual interior 3610 of a car from theinside.

IV.U. EMD Aware Industrial Design Display

Such a workspace for virtual parts design is shown in FIG. 37, reference3700, where an engineer 110 in a cubicle 3710 is wearing an EMDS 105,and is visualizing and modifying the design of a crank shaft 3740. Atracker frame 230 and a wireless pseudo cone pixel data streamtransceiver 228 are built into the upper portion of the cubicle 3710.

IV.V. EMD Aware Telepresence Display for Remote Teleconferencing

EMD Aware: Modifying the EMDS Scaler HW, Resolution, Wide Field of View,Stereo, Head-tracker, Augmented Reality, Virtual Reality, Eye-Tracker,Pseudo Cone Pixel Data Stream

Users wearing an EMDS 105 and in range of a network connected pseudocone pixel data stream transceiver 228 (and tracker frame 230) can bepart of a virtual teleconference. FIG. 38, reference 3800, shows theview from one physical participant 110 and two virtual participants,3810 and 3820. Each is wearing an EMDS 105, and have the same threedimensional data about the object 3740 that is the topic of theteleconference.

IV.W. EMD Aware Augmented Display for Equipment Repair

The prototypical example of using augmented reality for equipment repairis shown in FIG. 39, reference 3900. A technician 110 is half wayimmersed in a jet engine 3910 under repair, wearing an EMDS 105, andpulling up appropriate overlay schematics and instructions on a virtualdisplay 730 aligned via a physical world tracker frame 230 bolted to aknown part of the engine.

IV.X. EMD Aware Industrial Virtual Reality Display for SoftwareDevelopment in a Cubicle

Software engineers 110 can take advantage of EMDSs as well, even with noexplicit 3D content. Such displays allow them to have multiple webpages, documentation pages, code pages, and a debugger display either asseparate windows on a single cylinder virtual display 730 as seen inFIG. 40, reference 4000.

IV.Y. EMD Aware Industrial Immersive Virtual Reality Display in aCubicle

Just because an employee is housed in a small cubicle, does not meanthat they can not perform work on very large virtual spaces. A gamedeveloper can black out all real world light coming into his eyes byplacing (or pulling down) blackout shades 4110 over the EMDS headpiece.Now a fully 3D stereo rendered world can be displayed wherever the gamedeveloper looks, otherwise using the same set-up as was shown in FIG.40.

IV.Z. EMD Aware Telepresence Display for Remote Medicine, Robots, Land,Sea, and Air Vehicles, Space, Planetary Explorations (Moon, Mars, etc.)

EMD Aware: Modifying the EMDS Scaler HW, Resolution, Wide Field of View,Stereo, Head-tracker, Augmented Reality, Virtual Reality, Eye-Tracker,Pseudo Cone Pixel Data Stream

EMDS 105 can be used as part of a “telepresence” system, where theviewer 110 and the object being viewed are separated by a (large)physical distance. Most existing telepresence systems are based onlimited resolution non-stereo standard television systems, and are thuslimited in their application. Assuming stereo or multiple cameras at theremote end, much more “real” remote viewing can be achieved utilizingEMDS as the display component. Also, if the time delay loop is shortenough, then the camera systems at the remote end can have higherresolution “foveal” centers than the rest of the view. Then the eye (andhead) tracking data from the EMDS can be used to point the remote fovealcameras in the appropriate direction so as to maximize the resolutionsent back to the remote viewer 110.

Down to earth systems are remote robots where it is dangerous,impossible, or too time sensitive for a person to go, as well as systemsjust to lower travel costs and times such as remote medicalapplications.

Outside the earth, even if one has a nice Mars base and a nice crewedexploration vehicle, a (possibly tethered) robot with stereotelepresence cameras can still be a better “crew member” to go outsideand investigate some interesting rocks. Limiting the number of occasionsthat the human crew has to “suit up” and enter the hostile off-earthenvironments reduces the risks of accidences and increases productivity,not to mention that the robot can see in many spectrums that theun-aided human eyeball cannot.

FIG. 41, reference 4100, shows a similar example for an EMDS 105 wearingastronaut 110 working in a short sleeves environment inside a spacestation 4120 is controlling via telepresence a robot 4130 working atsome physical task out in the vacuum beyond the station's walls.

V. Eye Mounted Displays and Eye Mounted Display Systems

V.A. Optical Basis for Eye Mounted Displays

The following discussions use the wavefront interpretation of light.Specifically, most natural objects (and most traditional displays), froma light propagation point of view, consist of physical surfaces where atlarge numbers of different positions on the physical surface pointsources of light exist generating spherical wavefronts of light. Theoptical frequencies (i.e., wavelengths) of this reflected lightcorrespond to the optical frequency of illumination light hitting thephysical surface in a region containing the point source. Thisdescription is a simplified model sufficient to illustrate the points tobe made. More detailed models can include additional effects such assubsurface scattering, polarization, frequency shifting, etc.

FIG. 11A shows an example two-dimensional cross section of a surface,such as the face of a rock cliff wall 1110, with only one point sourceof reflected/scattered light 1120 and its expanding wavefronts drawn,along with a human observer 110. In the natural and built environment,most such point sources are not self-emissive, but reflections of asmall portion of a larger illumination source, such as the sun, moon,fires, artificial lighting, etc. There are only a few other naturalself-emissive light sources, such as bioluminescence. The expandingwavefronts of light, such as from point source 1120, are what the humaneye is designed to convert into images on the surface of the retina, aswill be described later. But first, a description of how existingdisplay technologies form similar sets of wavefronts of light will beconsidered.

In contrast to the natural environment, most direct view displaytechnologies are self-emissive, including direct view CRTs, most LCDs,plasma, LEDs, OLEDs, etc. The few exceptions include reflective displaysthat emit no light themselves, but selectively reflect externalillumination sources. Projection displays are a specialized type ofillumination sources, where at an external in-focus image plane (i.e.,the screen), different small areas of the screen (individual pixels1160, or similar objects) are each illuminated by an independentlycontrollable intensity (gross number of photons per time period) and oneor more of specific spectral profiles (colors). This is achieved by theprojector emitting collapsing spherical wavefronts in a differentpropagation direction per “pixel” (or similar object). The optics areset up such that at a specific distance from the projector, all of thesecontracting wavefronts have contracted to very close to their minimumsize, preferably each non-overlapping each other, except for multiplespectral contributions (for example, red, green, and blue pixelcomponents all on collapsing to the same small area) forming a twodimensional array of these concentrated wavefronts. Almost all theprobability of each original truncated spherical wavefront emitted fromthe projector has been concentrated into these individual small areas,concentrating the probability of the wavefront eventually collapsinginto a photon to each individual small area. Only some wavefrontscollapse into photons at the screen; these are absorbed by atoms in thescreen, and are generally converted to heat. But in most cases thecontracting wavefront is reflected or scattered (sometimes severaltimes) by atoms in the screen, thus changing the incoming collapsingwavefront into multiple new point sources of expanding spherical wavesfrom different points 1170 within the macroscopically small area, asshown in FIG. 11B. This collection of expanding wavefronts from thescreen surface approximate the collection of expanding wavefrontsproduced in natural conditions, and as will be described in a latersections, allow the natural function of the human eye to perceive theseartificially generated collections of expanding wavefronts as images.

FIG. 86 shows multiple wavefronts 8610 emitted by the point source 8600.While the wavefronts are initially spherical, in FIG. 86 the wavefronts8610 are eventually truncated to show only those portions that will passnear the human eye. As can be seen in FIG. 86, only those portions ofthe wavefronts 8610 that intersect with the cornea will enter the eye(ignoring reflections off the cheeks, etc.). As the wavefronts 8610 passthrough the cornea, their shape will be changed. The exact nature of thechange in wavefronts 8610 shape is a function of corneal shape, theshape of the wavefronts 8610 as they encounters the cornea (usuallyportions of spherical wavefronts of a given radius), and the specificoptical frequency of the emitted wavefront 8610. This function can besimulated by computer programs. See, for example, U.S. patentapplication Ser. No. 11/341,091, “Photon-Based Modeling of the Human Eyeand Visual Perception,” filed Jan. 26, 2006 by Michael F. Deering, whichis incorporated herein by reference.

In general, though, the wavefront modification caused by the cornea isto change the wavefronts 8610 from expanding wavefronts to contractingwavefronts. As seen in more detail in FIG. 87, the modified wavefrontsare post corneal wavefronts 8710. These wavefronts propagate through theaqueous humor until they encounter the (variable size and distance)iris. Only those portions of the wavefronts that intersect with the holein the iris will pass through the pupil and enter the lens. Thesewavefronts are the post pupil wavefronts 8720, which are a truncation ofthe post corneal wavefronts 8710. The lens will perform additionalmodifications to the wavefront 8720 to produce the post lens wavefronts8730. The wavefront shape change performed by the lens is again afunction of present shape of the variable shape lens, the incoming postpupil wavefronts 8720 shape, and the specific optical frequency of thepoint source 8600. This function can also be simulated by computerprograms. See U.S. patent application Ser. No. 11/341,091, cited above.In general, though, the wavefront modifications caused by the lens areto further reduce the radius of contraction and direction of propagationof the post corneal wavefronts 8710. These wavefronts 8710 propagatethrough the vitreous humor until they encounter the photosensitiveretinal surface.

Formally, the result is a probability distribution on the retina that isthe point spread function of the image of the point source 8600 on thephotosensitive retinal surface. While the tail of these functions canextend quite far, normally only a sub-portion of the retina thatcontains a large majority (say 95%) of the probabilities is identifiedas the illuminated photosensitive retinal surface portion (for opticalfrequency of the point source 8600). If the distance from the pointsource 8600 to the eye at the optical frequency of point source 8600 is“in focus” at the photosensitive retinal surface, then the portion ofthe probability of any point on the wavefront collapsing to a photonwill be focused on a particular small portion of the photosensitiveretinal surface.

In the fovea, the point spread function of the focused wavefront on aparticular point on the photosensitive retinal surface will bedetermined by a combination of the quality of the cornea and the lens asoptical elements, and the diffraction effects generated by the size ofthe pupil. Within the region of the fovea, this point spread functioncan have the majority of its probability contained within an area notmuch larger than a single thin foveal cone, but the higher the retinaleccentricity the larger the point spread function will get, due mostlyto the imperfect nature of the human eye's optical elements.

Considering together all these operations, it can be seen that twodifferent point sources of light, positioned at different angles inspace, will concentrate different photon collapse probabilities tospecific different illuminated photosensitive retinal surface portions.A first point source will be imaged on the retina at one retinal imagepoint a second different point source will be imaged on the retina at asecond different retinal image point. By adding more and more angularlyseparated points, one can see how the human eye produces an (inverted)projected two dimensional image of the three dimensional environmentaround it onto the (approximately spherical) photosensitive retinalsurface.

FIGS. 86 through 91 and 42 through 48 illustrate optical properties ofthe human eye that will be later used to enable the construction of eyemounted displays. In these examples, a point source generates awavefront, a portion of which passes through the cornea and is imagedonto the retina. The wavefront changes as it passes through the eye.This function can be simulated by computer programs. See, for example,U.S. patent application Ser. No. 11/341,091, “Photon-Based Modeling ofthe Human Eye and Visual Perception,” filed Jan. 26, 2006 by Michael F.Deering, which is incorporated herein by reference.

FIG. 86 was described above. FIGS. 88 through 91 are modifications ofFIG. 86. In FIG. 88, the portions of the wavefront 8610 that will notencounter the cornea are drawn as dotted lines 8800; the portions of thewavefront 8610 that will have their shape modified by the cornea to thewave front 8710 but will not encounter the pupil are drawn as dashedlines 8810; and the portions of the wavefronts 8610 that will make itall the way to the photosensitive retinal surface and produceillumination on the photosensitive retinal surface portion are drawn assolid lines 8820.

In FIG. 89, only the portions of the wavefront that will make it to thephotosensitive retinal surface (the solid portions of FIG. 88) andproduce illumination on the photosensitive retinal surface portion areshown, along with a thicker line outline showing the (one dimensionalcross section of the) envelope of this truncated wavefront. The fullythree dimensional envelope is the optical aperture of a retinal area8900, which looks like a three dimensional ellipsoidal cone with somebends in it. In FIG. 89, only the two dimensional cross section of thisthree dimensional object is shown. Both are identified as reference8900.

In FIG. 90, the portions of circular arcs representing the wavefront atdifferent locations are no longer drawn, leaving only the (twodimensional cross-section) optical aperture of a retinal illuminationenvelope 8900 to show the boundaries of the wavefront that will make itto a retina area and produce illumination on the photosensitive retinalsurface portion. The portion of the front surface of the cornea that iswithin the optical aperture of the illuminated photosensitive retinalsurface portion is indicated by drawing that portion of the frontsurface of the cornea as a thicker line 9000 than the rest of the frontsurface of the cornea. The retinal illuminating corneal sub-surface 9000is formed by the intersection of the optical aperture of the illuminatedphotosensitive retinal surface portion with the surface of the cornea.The prefix “sub” in “corneal sub-surface” refers to the fact that thisarea is a subset of the full corneal surface and does not imply thatthis is necessarily below the corneal surface. In general, its edgeshape resembles an ellipse cut out of the roughly parabolic surface ofthe cornea. The two dimensional cross section of this sub-surface isreference 9000 in FIG. 90.

This illustrates an important aspect of EMDs. Conventional displaysgenerate wavefronts of light that cover at least the entire cornea andnearly always much more. However, to illuminate a particular smallportion of the photosensitive retinal surface, one does not need togenerate relatively large area wavefronts of light, as is done inconventional displays, where the wavefront area has been at a minimumthe size of the eye, or much larger. Instead, it has been shown herethat for a display positioned outside the cornea, one need only generatewavefronts that cover the respective retinal illuminating cornealsub-surface, whose area is considerably smaller than the entire cornealarea. That is, the pupil acts as an aperture. The projection of aparticular photosensitive retinal surface portion through the pupil ontothe cornea defines (at least to first order) an area on the cornea thatwill be referred to as the retinal illuminating corneal sub-surface, orsimply the corneal aperture, for that particular portion of the retina.This effectively is the projection of the optical aperture onto thecornea. Wavefront portions (of the correct wavefront shape) that fallwithin the corneal aperture will propagate on to the correspondingphotosensitive retinal surface portion. Wavefront portions that falloutside of the corneal aperture will be blocked, for example by opaqueportions of the iris.

Note that any wavefront that is smaller than but still within thisretinal illuminating corneal sub-surface (and with the correct wavefrontshape) will also illuminate the same photosensitive retinal surfaceportion. This situation will be referred to as an underfilled cornealaperture. Note that the pupil will also be underfilled in this case. Onedrawback of wavefront portions that do not fill the corneal sub-surfaceis that the diffraction effects are larger, but outside the fovea regionthis is rarely the resolution limiting effect.

FIGS. 42 through 44 are three dimensional illustration of the pointsmade above. In these Figures, the eye is the right eye and the pointsource 3500 is assumed to be off to the right of the person. Features ofthe face are shown in order to better show the changing threedimensional perspectives. In FIG. 42, the point of view is from thepoint source 3500 looking straight at the pupil 1430.

In FIG. 43, the point of view is half way between the point of view ofFIG. 42 and a point of view that is head-on to the face. We now see inthree dimensions the corneal aperture 3900 from this different angle.

FIG. 44 is from a point of view now looking head-on to the face. We nowsee the corneal aperture 3900 from more fully as the intersection of acone with the cornea at an even larger angle in three dimensions.

Using a three dimensional model of the optics of (truncated) wavefrontsof light from a point source of light in the external environmentpropagating through the optical elements of the eye, it has been shownthat only a truncated wavefront covering only a small portion of thecornea 3900 will be the only external wavefronts that will eventuallyreach the small portion of the photosensitive retinal surface thatimages that point source (for reasonably focused conditions of the eye'soptics relative to the external point source).

In turn, this proves that an eye mounted display need only generatewavefronts from a particular direction of propagation whose envelopesintersect a subset of the corneal aperture 3900 for each small region onthe photosensitive retinal surface that the display wishes to form apixel or similar object on, and still have the ability to form arbitraryimages on the photosensitive retinal surface. Using these smallercorneal regions for display results in many advantages. As will bedescribed in more detail later, miniature display devices that aresub-parts of an EMD can be made considerably simpler and smaller thanprevious art displays that had to generate a significant portion of theentire image to be presented to the user's eye. As one example, they infact can be made so small as to fit within a modified contact lens. Inother examples, the display can be placed within the eye itself. Anotheradvantage is a significant reduction in the amount of light that must begenerated to form reasonably bright photopic images to a human 110viewer. Many other advantages are described elsewhere in this document.

For a given eye, with a given radius pupil, and given lensaccommodation, for a given receptive field center (the desiredilluminated photosensitive retinal surface portion), there exists aunique corneal aperture 3900 that will “address” this receptive fieldcenter. The job of an eye mounted display external to the cornea is togenerate the properly shaped optical wavefronts and entry regions of thecornea to produce regions of photosensitive retinal surface illuminationwhose point spread functions are close in size to the size of thereceptive field centers that are in the location of the photosensitiveretinal surface (or smaller in some cases).

It should be noted that in nature, in the high resolution foveal region,it is not possible to produce spots of retinal illumination that enteronly a single cone. Point sources of light outside the eye will generatespots of illumination that at a minimum will also enter the first layerof cones surrounding any specific cone, though at reduced brightness. Itshould also be noted that such small spots as were just describedcorrespond to 20/10 vision, which only a small portion of the populationhave. The more typical resolution of the general population is in therange of 20/18 to 20/30. In terms of eye mounted displays, this meansthat the resolution limit for most of the population can be reached bydisplays whose smallest point spread functions generatable could be aslarge as four foveal cones (assuming the smallest cones of persons with20/10 vision—most people have cones that are 2× or more larger at theirsmallest, or have equitant resolution limits in their eye's opticalpath). This larger limit will become important when discussingmanufacturability of embodiments of specific designs of eye mounteddisplays.

The same analysis can be performed for the larger receptive fields ofrods; but because in most ways such an analysis would be a sub-set ofthat performed for cones (except for dealing with significantly lowerlevels of light), and from the teachings given here, is easily derivedby one skilled in the art, an analysis of the equitant for rods need notbe expressly presented here.

The same analysis can be performed for eye mounted displays that produceoptical wavefronts at locations within the human eye's optical pathother than above the cornea. From the teachings given here, thesealternative displacements can be derived by one skilled in the art.Accordingly, an analysis for all the other possible locations of lightemission will not be presented here.

The layers of neurons between the output of the photoreceptor cones andoutput of the eye, the optic nerve, perform a plethora of differentprocessing computations on the cone output data. For the purposes ofthis disclosure, a simplified model of most of the data output from theeye, cone retinal receptive fields, is sufficient. Accurate models ofcone retinal receptive fields are important to eye mounted displays intwo ways. First, they change in size and their size as determined byboth retinal eccentricity and co-latitude establishes the maximumresolution in a particular sub-region of the retina that the eye mounteddisplay needs to generate for that sub-region if maximum resolution isto be achieved. Second, an eye mounted display does not have toprecisely duplicate the illumination pattern on the retina as whatnatural world produces for a similar visual scene. The more importantgoal is through illumination of the retina to cause the retinalcircuitry to as closely as possible replicate the computed output signalgenerated by the cone retinal receptive fields.

An abstract model of a retinal receptive field includes two differentretinal receptive field sub-fields: the retinal receptive field centerand the retinal receptive field surround. Three mechanisms cause theretinal receptive field center (eye pixels) to vary in area. First, thehead-on area of cone cells is the smallest at the very center of thefovea. At one degree of visual eccentricity away (the edge of thefovea), the area of cone cells may have doubled or tripled. The area ofthe cone cells continues to increase with greater visual eccentricity(with some additional variation in visual co-latitude) all the way outto the ora serrata (though the rate of growth greatly slows at abouthalf way to this edge). The area between cone cells, which hardly existsin the packed center of the fovea, also grows with greater visualeccentricity as smaller rod cells start intermingling between the conecells. The other cause of increase in retinal receptive field centersarea are due to the change in nature of the retinal receptive fieldcenters from being just a single cone cell at the center of the fovea,to the retinal receptive field centers being formed by larger and largergroupings of cone cells at increasing eccentricity.

All three of these effects are shown in FIG. 91, reference 9100.Reference 9110 shows how retinal receptive fields are formed from conecells 9110 at 0° of retinal eccentricity (the center of the fovea).Reference 9120 shows how retinal receptive fields are formed at 0.9°(outer edge of fovea, and edge of the region where the center is asingle cone). Reference 9130 shows at 9° (example of center beingcomprised of multiple cones). All three fields are drawn using the samephysical scale, with element 9140 showing ten microns for reference.These are all “center on” fields. The symmetrical “center off” fieldsexist at the same location (generally) using the same cones, but withinverted signals before summation and thresholding before transmissionout of the optic nerve.

Because the optics of the eye degrade at larger and larger visualeccentricity, the actual area of a cone cell is not so important. Whatis important is the density of cone cells at a particular visualeccentricity (and co-latitude). Conventionally this density is measuredin units of number of cone cells per square millimeter (with the eyeradius normalization convention discussed earlier).

Thus if a designer of an EMD wants to know what size “eye pixel” wouldgive the best resolution in a specific region of the retina, he can lookup the retinal cone density for that region, invert the density toestimate the average area of a cone cell and its share of the areabetween cone cells within that region, and then multiply that area timesthe number of cone cells that comprise the retinal receptive fieldcenters within that region. He can convert between retinal area andvisual angle as needed for other uses. These location specific cone celldensity numbers are available from a number of sources in theliterature. For example, see Curcio, C.; Sloan, K.; Kalina, R.; andHendrickson, A.; “Human Photoreceptor Topography,” J. ComparativeNeurology 292, 497-523 (1990); Tyler, C., “Analysis of Human ReceptorDensity,” in Basic and Clinical Applications of Vision Science, Ed. V.Kluwer Academic Publishers, 63-71 (1997); and as in U.S. patentapplication Ser. No. 11/341,091, “Photon-Based Modeling of the Human Eyeand Visual Perception,” filed Jan. 26, 2006 by Michael F. Deering; allof which are incorporated by reference herein. The number of cone cellsthat are grouped together in the retinal receptive field centers for thecan be estimated from spatial frequency studies of the region inquestion.

The size of the receptive field components at greater eccentricitiesgrow in size even faster than the distance between cones grows. Thisexplains why although the human eye contains more than five million conecells, it only contains 800,000 retinal receptor fields and as half ofthose are duals of each other. Thus, there are only 400,000 uniqueretinal receptive field locations for the entire retina. This spatialvariable resolution by eccentricities has been confirmed by manydifferent experiments, including physiological experiments (eye tests atdifferent eccentricities). Thus an eye mounted display need only controllight aimed at these 400,000 unique retinal receptive field centers,which becomes a progressively easier job outside the fovea, as the sizeof the receptive field centers become fairly large.

It can be noted that the 800,000 unique retinal receptive fields per eyeis supported by the fact that the optic nerve (leaving the back of theeye into the rest of the brain) is comprised of only one million neuralfibers and at least 200,000 of them are doing other things thantransmitting retinal receptive fields results. It can also be noted thatthe number of display pixels needed to form the highest naturalresolution image on the retina (and thus the cones) is not necessarilyone-to-one. Better to perfect coupling between the display and theunique retinal receptive field centers can require that the displaypixel count is larger by a small multiple. However there is adiminishing return in perceivable quality to the human viewer withincreased pixel density too much past the retinal receptive fieldcenters density. Other factors, such as optical blur and chromaticaberration of the eye's optical elements, coupled with diffractioneffects sets the limits in display pixel density. For simplicity, mostof this document assumes a particular sub-set of EMDs in which the twodensities are the same but this is not intended to limit the scope ofthis work.

V.B A New Approach for Display Technologies

Nearly all previous existing display technologies emulate opticalreality at a level some distance away from the cornea. They generatespherical wavefronts with diameters at observation covering anywherefrom several thousand feet (in a sports stadium display), to a dozenfeet (home HDTV screen), to less than an inch, for the special case ofinstruments with a narrow entrance pupil for the observer's eye (e.g. amicroscope or telescope eyepiece, and most head mounted displays). Thevast majority of computer and television displays in use today arewithin the tight range of a foot to a few feet wide. At normal viewingdistances, the radii of the spherical light wavefronts generated areapproximately on the same order of size.

In contrast to existing display technologies, the display technologydescribed below reduces the light emitted for a given pixel (or equitantobject) to the retinal illuminating corneal sub-surface 3900, or aworkable subset of this area (i.e., an underfilled corneal aperture). Intheory, a display device generating a wavefront that covers the cornealaperture 3900 for every retinal center-surround receptive field centerarea in the eye, would be able to match the eye's perception of almostany physical world scene. The device would be able to synthesize nearlyany image at the same resolution that the eye can perceive.

An eye mounted display constructed to generate a number of wavefrontsdirected to different corneal apertures 3900, whose point spreadfunction on the photosensitive retinal surface is at the approximatesize, density, and shape as the retinal receptive field centers in thelocal vicinity of the addressed portion of the retina, but perhaps notexactly matched to the individual retinal receptive field centers of aspecific eye, can generate a high quality and large field of viewdisplay. In fact, because the display is not locked to any specificretinal optical reception areas, a number of real-time corrections(warping, etc.) to the image can match other parameters (such asaccommodation, or slip in coupling) changing. Also, consider that due todrifts, in the real world point sources of light are rarely imaged by asingle cone. Instead a slightly blurred retinal image is spread acrossand sensed by two or more retinal center-surround receptive fields.

Consider a display device that generates, for a given desireddistribution of spot sizes and locations on the photosensitive retinalsurface, the corresponding full corneal apertures 3900. Then if onedraws the outlines for all these apertures, they would overlap togreater or lesser extents a large number of other nearby apertures andthere would be no way to partition the apertures into disjoint groups.In some embodiments, this is not a problem, and the appropriate radiusexpanding wavefronts of light from the appropriate directions aregenerated by and EMD truncated into all the appropriate cornealapertures 3900.

However, for other embodiments, it is more convenient if the cornealapertures 3900 generated can be partitioned into differentnon-overlapping groups. This is not possible if one wishes to fill eachentire aperture. However, it is possible if one accepts a little moreresolution loss due to diffraction. If in place of the full area cornealapertures 3900, instead (for example) a quarter area aperture of eachcorneal aperture 3900 is generated, such disjoint partitioning ispossible. In other words, the pupil is underfilled. In this case, theless than full corneal aperture will be referred to as a cornealsubaperture or an underfilled corneal aperture.

To see how a disjoint partitioning is possible, first note that thecorneal quarter-aperture (i.e., a subaperture that is a quarter of thearea of the full aperture) can be placed anywhere within the fullaperture 3900 and still generate a spot of light at the same position onthe photosensitive retinal surface. Next, note that if the position ofthe quarter-apertures can be biased toward one side of the correspondingcorneal full-aperture 3900 in the direction of a local center point,then when all the quarter-apertures are drawn on the cornea, they canform disjoint sets around each local “center” point.

As a vastly simplified example to illustrate the point of the lastparagraph, consider a retina that only has nine cones. FIG. 45,reference 4500, shows a diagram of the cornea for this simplified eye.Element 4505 is the outer extent of the cornea, as seen by orthographicprojection down the optical axis of the cornea. Each of the nine coneshas a corresponding corneal aperture, which are represented by thereferences 4510 through 4550, respectively. The positions of 4510through 4550 shown correspond to the center of each corneal aperture. A3 mm virtual entrance pupil was used in this computation. The cones areat a visual angle of 26.6°, and equally spaced around 360° with 40°between each.

In FIG. 46, the edge of each corneal aperture has been added as thereferences 4605 through 4645, respectively. In other words, the cornealaperture for cone 1 is defined by the boundary 4605, which is centeredat 4510. Note that even in this simplified example, the cornealapertures significantly overlap. However, as shown in FIG. 47, if oneuses a display extent of less than the full aperture size, onesub-display 4700 can be used to address three separate cones whosecorneal apertures are shown in solid lines: 4605, 4610, and 4615. Theother six cones are shown in dashed lines for context. Note that eventhough the sub-display 4700 covers some of the corneal aperture of theseother cones, no light will fall on any of these so long as thesub-display 4700 only generates wavefronts of light that focus on one ofthe targeted three cones. In FIG. 48, it is shown how three sub-displays4700, 4810, and 4820 can address all nine cones.

Clearly we want a display that can address more than nine cones. But theoptical properties for any number of cones operate in the same manner.Given a contiguous region of the retina for which one wants to generatea display, one can take the intersections of all the optical aperturesat the retinal surface from all the cones in the region. So long as theregion is convex, the same result can be achieved by taking theintersection for the cones on the boundary edge of the region.Furthermore, for the double truncated circular pie wedge (which is anadvantageous shape to have a given sub-display display to), taking theintersection of the four cones at the four corners of the region cangive the correct result. Given some quantization on the incremental sizeof a sub-display region by the receptor field center sizes, and anyother desired constraints, exhaustive computer simulations of allpossible numbers of, positions of, and sizes of, sub-display can besimulated, allowing one to optimize the design of sub-displays of an EMDto any desired constraints (so long as a solution exists).

One such constraint could be that the addressed portions of the retinaby each sub-display slightly overlap all its neighbors. The overlaps canbe “feathered” together, employing any of several techniques that havebeen used in the past with (much larger!) multiple projector displays.

In one embodiment, these sub-displays would be femto displays.

It is important to note that diffraction effects of employing a quarter(or other partial) corneal aperture verses a full area corneal aperturecorrespond to the diffraction limits of approximately 20/20 vision vs.20/10 vision. As most people have closer to 20/20 vision, and relativelyfew are close to 20/10, the quarter area compromise will cause only aminor reduction in resolution over the best that they can perceive. Thisis an acceptable trade-off for many embodiments of EMDs.

We have now described at a high level the physical effects used to buildmany different embodiments of eye mounted displays. There are manyembodiments for devices to produce multiple specified radius expandingspherical wavefronts of light of a specific frequency (or frequencyspectra), propagating in a specific direction, and entering the cornealsurface within a specific truncated outline (i.e., partial cornealaperture). One class of such examples is embodiments of femto displaysas previously defined. This particular class of sub-display embodimentswill later be used to describe more details of a complete EMD and EMDS105. From this description it can be seen how such devices can be builtwith other embodiments of the sub-displays, or possibly using just onedisplay.

V.C Sub-Displays

The function of a sub-display is to generate the appropriate opticalwavefronts for the corresponding retinal region. Typically, thesub-display will be able to generate many approximately sphericalwavefronts, at slightly different directions of propagation, in oneembodiment, all truncated by approximately the same outline within andsmaller in area than the full area corneal aperture for the directionsof propagation. In the case of spherical wavefronts, the radius of thespherical wavefronts produced could be controlled per wavefront or, in asimpler embodiment; they could all have the same pre-set radius. Suchfixed radii would produce images that are in focus only for one focusdistance of the crystalline lens (but which is also a fixed parameterfor older people with presbyopia). A slight difference between the fixedradii of the sub-displays allows the surface of focus to be flat,cylindrical, spherical, etc. The collection of wavefronts produced froma particular direction over a time frame (for example, the time of oneframe of display) has a statistically controllable intensity, as well asa statistically controllable mix of optical frequencies (color). If thesub-display embodiment is not much larger than the outline within thearea where wavefronts of light are produced, this could allow asignificant amount of normal external physical world produced light topass through the cornea normally, thus producing a “see-through”display. In addition, if partially silvered front surface mirrors areused for the final optical element of the sub-display (as describedlater), then external light can come in throughout the EMD, just at areduced intensity (which is desirable for limited output intensityEMDs).

So far the discussion has concentrated on embodiments of EMDs thatproduce light wavefronts outside the cornea, with an air gap between theEMD and the cornea, or an air gap between the EMD and a corrective lensthat may be coupled to the cornea by tear fluid. This was done to makeexplicit the direct match between wavefronts of light in the physicalworld and the wavefronts of light produced by the new displaytechnology. However, the definition of EMDs includes those in which thedisplay can be placed on and/or in multiple locations within the eye.For these cases, the same sort of backward examination of modified lightwavefronts from where the display elements are placed, on and/or withinthe eye, to the world outside, will describe the modified wavefronts oflight that the display must produce to match how light wavefronts fromthe physical world would be modified at that point(s) on and/or withinthe eye. One simple example is an EMD in which the EMD is placed in amodified contact lens, with an air gap below the display and theposterior surface of the corrective contact lens. Now the matching taskis to match the wavefronts that the contact lens, rather than thecornea, would normally “see” from the outside physical world. In otherembodiments of EMDs placed further within the eye, the principle of“matching” wavefronts would be the same, but the wavefronts produced bythe display can be quite different.

The description of all the parameters to be taken into account in orderto produce each wavefront from the EMD that nearly exactly emulates aspecified point source in the outside physical world can be fairlystraight forward. In embodiments that only emulate fixed distances offocus, the position of the eye's lens will be known due to eye tracker125 and/or head tracker 120. With near cone accuracy tracking of theorientation of the cornea relative to the head (or some other knowncoordinate frame) by the combination of eye-tracking and head trackingdevices, the small target area of the retina that each wavefront(truncated to or within the appropriate outline) will be know, and canbe used to determine what intensities and colors should be displayed byeach separate wavefront generator (i.e., each sub-display).

V.D Embodiments of Contact Lens Mounted Displays

One sub-class of eye mounted displays is cornea mounted displays (CMDs).One sub-class of cornea mounted displays is contact lens mounteddisplays (CLMDs). One sub-class of contact lens mounted displays (CLMDs)is modern sclera contact lens mounted displays (SCLMD). The discussionbelow will use a particular embodiment of SCLMDs as a concrete exampleof a complete instance of an EMD, but will also discuss more generalCLMD issues.

When a contact lens is worn, most of the light bending now occurs in thecontact lens, and now very little light bending occurs in the cornea.The proper wavefronts for the sub-displays to generate are now thoseexpected at the surface of the contact lens, not at the surface of thecornea. This assumes that the contact lens is coupled to the cornea bytear fluid, and the sub-display has an air gap between its posterior andthe anterior of the optical zone of contact lens. In some cases theoptical zone of the contact lens is smaller than the field of view ofthe eye. In this case a vignetting of the eye's view will occur. This isa property of the contact lens. A contact lens with a suitably largeoptical zone will not have this limitation.

A relativity new type of contact lens is a hybrid of a soft large scleralens for contact with the eye, and a small hard lens in the optical zonefor vision correction. The sclera lens has a large amount of tear fluidbeneath it. This reduces the physical contact of the appliance with thesensitive cornea and also allows the natural nutrients and wasteproducts to be carried as normal by the tear fluid, which has a meansfor ingress and egress from the sclera contact lens. Because the scleralens is large, it is possible for it to be quite thick (1.2 mm or more)in the center of the contact lens. Because the change in thickness isgradual, the only part of the eye that might notice the extra bulge, theeye lid, usually is not bothered by this. In the thick center of thesoft sclera lens a cylindrical hole of soft lens material is removed,and a small hard contact lens is placed in. Because with the tear fluidthere is little change of index of refraction from the bottom of thehard lens past through the cornea, the primary optical bending takeplace at the air-hard lens boundary on the front of the hybrid contactlens. Because the corneal lens effectively does not contribute to theoptical function, any astigmatism (due to toroidal deformations of theeye extending to the cornea) can be effectively eliminated. The largesclera lens also does not move or rotate much, unlike more traditionalcontact lenses that can move up and down by their entire diameter duringeye blinks to allow an exchange of tear layer to take place.

One embodiment of a CLMD is as a modified form of a modified scleracontact lens (SCLMD). The idea is to place a display device (or set ofsub-display devices) in the cylindrical hole where the hard contact lenshad been, and optionally also place a thinner hard contact lens underthe display if ophthalmological correction is needed. It is usuallyimportant that there is an air interface between the bottom of thedisplay device and the top of the hard contact lens (if present) forproper functioning of the hard lens.

In one approach, as described above, the display task can be sub-dividedto a number of sub-displays, each emitting a number of sphericalwavefronts into their own particular partial corneal aperture. Manypractical solutions to the multiple non-overlapping projector placementproblem results in approximately 40 to 80 sub-displays using the samenumber of disjoint partial corneal apertures on the surface of thecornea or contact lens. These input regions will only cover about onefourth of the total surface area of the cornea or contact lens (orless), so the resulting optical system can have high quality see-throughvision of the natural world. For the present purposes, for now let usassume that the embodiments of the sub-displays are as femto projectors,and we will call the individual wavefront generating regions pixels. Nowturn to the details of implementing such femto projectors.

First a word about the pixels. In many embodiments it is more efficientto use hexagonal rather than rectangular shaped pixels, but many othershapes are possible. Also, like most direct view displays, rather thanbuild multi-color pixels, it is easier to assign each pixel to a singlecolor primary. However, unlike most direct view displays, the colorprimaries do not have to be equally represented or repeated. If threecolor primaries are used, targeting the optimal sensing frequency of thelong, medium, and short wavelength cones, the three primaries would bejust a variation of red, green, and blue. However, because the bluecones represent a ninth or less of the cones in the retina (and none inthe central most portion of the fovea), only one out of every nine“pixels” could be blue. Measurements of the ratio of red to green conesin the human eye have varied from 2:1 to 1:2. Thus, in one embodiment,the remaining eight ninths of the pixels are equally split between redand green cones (four out of nine each).

The abstract optical path for a femto projector can be simple. Place a128×128 (or so) image plane of pixels far enough away from a lens tocause the angle of each pixel relative to the lens to correspond to theinput wavefront angles desired over a particular patch of cones. Letthis angle be 2*n. The lens is a simple converging lens (positiveoptical power). It causes spherical wavefronts whose radius is only afew millimeters to appear to have a radius of (say) six feet. Asimplified two dimensional vertical cross section of such a femtodisplay 4900 is shown in FIG. 49, with the light direction indicated byreference 4940. The display source (array of pixels) is reference 4910.The half-angle 4920 that a pixel makes with the lens is n. Let thedistance from these display pixels (multiple point emitters of photonswithin the pixel active region) to the converging lens 4930 be d. Letthe height of the display pixels be h. For this femto projector toproduce light wavefronts subtending a half-angle of n the relationshipbetween h and d is:

$\begin{matrix}{d = \frac{h}{2*{\tan (n)}}} & (1)\end{matrix}$

In many implementations, d will be fixed, as will be n by definition fora given sub-region of the retina to be addressed, so for a particularfemto-projector h will then be fixed. As an example, a femto displaywith height h equal to 0.5 mm high and a desired spread angle n equal to10° yields a separation distance d of 2.9 mm.

Unfortunately, in the allotted space for the set of femto-displays, onthe order of a millimeter thick, there is not enough distance to placethe pixel displays directly in line with their converging lens. So wefold the optics. As shown in FIG. 50, a two dimensional vertical crosssection of a different femto display 5000, a 45° mirror 5010 allows oneto use lateral space on the display body to optically back up the pixeldisplays far enough from their corresponding lenses to obtain thedesired geometry. This figure shows the anterior 5020 and posterior 5030outsides of the contact lens capsule.

FIG. 50 shows the folded light path for one femto display. In a typicaleye mounted display, there may be 40-80 femto-displays, each with itsown folded light path. There are many different ways to let thesedifferent light paths cross through each other, and pack properly intothe desired volume. As shown in FIG. 51, it is also possible to combinethe lens and 45° turning mirror into one achromatic optical element 5110by reshaping the 45° flat mirror into a curved optical mirror thatperforms both functions, creating a femto display 5100. FIG. 52 is anoverhead view of the femto projector shown in FIG. 51. FIG. 53 shows anoverhead view of another femto display created by folding thefemto-display of FIGS. 51 and 52 in any of several different ways usingan additional folding mirror 5310. FIG. 54 shows how four femto-displayscan form a four times larger area synthetic aperture, making use ofseveral mirrors 5410, half-silvered mirrors 5420, 45 degree mirror andconverging lens 5430, and pixel display 5440.

FIG. 55 shows how an overhead mirror 5510 can make a long femtoprojector more compactly fit into the area between two parabolicsurfaces (such as within a contact lens), with the pixel display 5440one the left end and the 45 degree mirror and converging lens 5430 onthe right hand side.

FIG. 58 shows a human eye optically modeled in the commercial opticalpackage ZMAX. It contains a standard optical model lens 5810 equivalentto the human eye cornea, a standard optical model lens 5820 equivalentto the human eye lens and a standard optical model surface 5830equivalent to the human eye retina. FIG xx shows the results from ZMAXcomputing retinal spot sizes of this combined lens/surface system. Thesport sizes shown are comparable in size to the smallest human eyefoveal cones, so the optics has met its design goal.

FIG. 81 shows a vertical cross section of one example of afemto-projector. A 128×1 pixel bar of individually addressableultraviolet LEDs 8110 shines onto a MEMS oscillating UV mirror 8120,which reflects the line of UV pixels up and down across a 128×128 arrayof thin visible light phosphor pixels 8130. The output light directionis shown by arrow 8140. The relative placement of the elements is asimplified example. Many optimizations to the scanning are possible.FIG. 82, reference 8200, shows a perspective view of the display of FIG.81. While thin phosphor coatings can be illuminated by UV light frombehind (conventional CRT's are “lit from behind” phosphors), femtodisplays can also use phosphors lit from the front, as seen inhorizontal cross section in FIG. 83, reference 8300, and in 3Dperspective in FIG. 84, reference 8400.

To fit within the rest of the constraints, the shape of the hard contactlens containing the femto displays is thin (approximately 1.0 mm to 2.0mm in height) with spherical or parabolically curved outward top andinward bottom. We will call this the display capsule. In this design,the top of the display capsule forms a continuous surface with the topof the hybrid sclera contact lens, allowing the eye lids and eye lashesto smoothly pass over the surface, as shown in FIG. 65, reference 6500,in six time steps referenced from opened to closed to opened again:6510, 6520, 6530, 6540, 6550, and 6560.

The bottom is concave to keep the posterior surface at a near constantdistance from the cornea, and to allow an air gap between anophthalmological hard contact lens (if any) below the display capsule.The functional width of the display capsule preferably is at least thesize of the optical zone of the underlying hard contact lens, whichhopefully is at least as large as the primary optical zone of the frontindex of refraction modified cornea. The full width of the displaycapsule can be larger and the edges of the display capsule can be a goodplace for holding system component elements that do not emit light fortransmission to the eye. This specifically includes the possibilities ofEMD controller chip(s), batteries, camera chips and correspondingoptics, accelerometers, eye blink detectors, input power and/or signalphotodiodes, output signal transmission components from the EMD to theheadpiece, etc., as is shown in FIG. 78.

The outside shell of the display capsule should be as thin as possible,to keep from introducing optical effects of its own, but also hardenough to withstand the normal forces that any contact lens is expectedto take. There are several possible materials that can meet thisrequirement. One of them is vapor deposited diamond onto a mold. Thistechnology is presently used to produce inexpensive heat sinks, and tocoat the working tip of various cutting tools. A diamond display capsulecould be made in two halves. The rest of the active components placed inbetween the two halves, and then the two halves of the diamond capsulewould be hermetically sealed. There are also several special plasticmaterials now available that can be formed very accurately by molding.These have advantages over vapor deposited diamond. Both sides of eachside of the display capsule can be formed, and the rough inner side ofthe vapor deposited diamond does not have to be optically polished (at agreat cost). In some cases it may be possible to form parts of theoptical paths directly via the mold surface itself (e.g., though silverdepositing for mirrors may still be required) but most likely the innersides to the two display capsule molds will instead provide points ofattachment and calibration for separate optical and other components.

In FIG. 60, reference 6000, a perspective view of a complete assembledcontact lens display is shown attached to the human eye. In FIG. 61, anexploded view of the same contact lens display is shown as element 6100,containing the display capsule 6110, the battery 6120, and the scleralcontact lens body 6140.

FIG. 62, reference 6200, shows one layer of femto projector light pathswithin the display capsule. FIG. 63, reference 6300, shows a secondlayer of femto projector light paths within the display capsule. Thesetwo layers allow all femto projectors blockage-free light paths fromtheir phosphors to the corresponding fold mirrors that redirect thelight down through the contact lens and into the cornea. This is furtherdemonstrated in FIG. 64, reference 6400, a 3D perspective view of thecontact lens femto-projector light paths as viewed from under the lens.

As mentioned before, eye mounted displays can be placed anywhere withinthe optical path of the eye. The next several figures illustrate severalsuch different places. More that one of these may be used at the sametime. For example, an additional structure closer to the outside of theeye may be used for eye tracking purposes.

FIG. 66, reference 6600, shows a horizontal slice view of a contact lensbased eye mounted display 6610 in its natural environment—placed on topof the eye's cornea.

FIG. 67, reference 6700, shows a horizontal slice view of an eye mounteddisplay in which a display capsule 6710 is placed inside of or in placeof the cornea.

FIG. 68, reference 6800, shows a horizontal slice view of an eye mounteddisplay in which a display capsule 6810 has been placed on the posterior(rear) surface of the cornea.

FIG. 69, reference 6900, shows in horizontal cross section aconfiguration in which a display capsule 6910 is part of an intraocularlens, placed between the cornea and the lens within the anteriorchamber. This technique has several advantages over a contact lensdisplay. No contact lens need be put in and out of the eye. Ocularcorrection can be performed “traditionally,” either using exteriorglasses, contact lenses, or various forms of cornea surgery (e.g.wavefront LASIK) (or just via natural clear vision). In addition, thedisplay is positionally stable with respect to the eye and retina.

FIG. 70, reference 7000, shows in horizontal cross section aconfiguration in which a display capsule 7010 has been placed on theanterior (front) surface of the lens.

FIG. 71, reference 7100, shows in horizontal cross section aconfiguration in which a display capsule 7110 has been placed inside ofor in place of the lens.

FIG. 72, reference 7200, shows in horizontal cross section aconfiguration in which a display capsule 7210 has been placed on theposterior (rear) surface of the lens.

FIG. 73, reference 7300, shows in horizontal cross section aconfiguration in which a display capsule 7310 has been placed within theposterior chamber, between the lens and the retina.

FIG. 74, reference 7400, shows in horizontal cross section aconfiguration in which a display capsule 7410 has been placed close toor directly on the surface of the retina.

All of these examples simply represent single points among a continuumof possible ways of infiltrating artificial displays into the opticalpathways of the human eye. So far all of these techniques have onlydescribed simple cases in which a display capsule was placed at aparticular point within the optical path of the eye. This is not meantto preclude situations in which multiple artificial elements areintroduced to the eye (not necessarily into the optical path). Onespecific example is the situation in which calibration marks for eyetracking have been made directly on the surface of the scalia for areader that is tucked inside the eye orbit (and thus is cosmeticallyacceptable since nothing shows externally).

V.E Internal Electronics of Eye Mounted Display Systems

FIG. 75, reference 7500, shows one possible physical shape of aheadpiece 7510, modeled after a pair of sunglasses. Also shown in FIG.75 are the nose bridge 7520, the light occluding sides of the headpiece,and the left ear audio output 7540.

FIG. 76, reference 7600 shows a logical level example of the headpieceelectronics. The pseudo cone pixel data stream 225 input is reference7605. The rules for transmitting protected media content (like Blu-Ray™or HD-DVD™ video discs) require specific encryption when full fidelityimages are being transmitted. In all likelihood, the real-time variableresolution moving point of view pixel display frames will not be deemedto require encryption. However, the PCPDS information is preferablyencrypted, and may be decrypted at this point by a specific decryptioncircuit 7610. Although most of the time, reference 225 is described asdata flowing towards the eyes, in fact the channel 225 preferably isbidirectional, as calibration and other data can flow away from the eye,although probably with a lower bandwidth.

Reference 7615 and 7620 are the pseudo cone pixel data stream 225signals going from the headpiece to the left and right EMD,respectively. These carry the pixel information for each frame ofdisplay. The data rate for this information channel preferably is highenough to carry single component pixel information for around 500,000pixels every frame time, which can range from 50 Hz to 84 Hz or higher.Simple lossless compression techniques can be applied to thisinformation flow, so long as the decompression algorithm requires only asmall amount of computation. For relatively small field of view virtualscreens within the very wide field of view display, there can be a lotof blank pixels that even simple run-length compression will easilyhandle. But also remember that the fovea, where 10% or more of thedisplay pixels live, will be looking right at the small display, so theoverall compression will be smaller than with a non variable resolutiondisplay. Slightly lossy compression algorithms may be acceptable in manycases, especially if it is “visually lossless.” Fortunately “eye safe,”water penetrating, mid infrared frequencies can easily handle therequired data bandwidth, and at the safety-required low transmissionpowers. A portion of this infrared transmission can be picked up by oneor more photo diodes 7840, 7845 or 7850 tuned to the same infraredfrequency located just under the top of the display capsule, as is shownin FIG. 78, reference 7800. Because the eye rotation is tightly tracked,even lower power transmissions are possible if the transmission from theheadpiece closely tracks where the closest display capsule photodiode islocated.

Embedded DSP cores 7625 perform much of the data processing for theheadpiece, and since they are programmed, in a re-programmable way.Which portions of which computations are in dedicated logic versus theDSP is an implementation dependent choice, but it the eye and headtracking algorithms do require some amount of programmable computationalresource. The EEPROM 7630 (or some other storage medium) can contain allthe code for the DSPs 7625, as well as specific calibration informationfor a particular pair of EMDs. This information is downloaded to thescaler subsystems 202 through 210 during system initialization. In thisway, different people can plug into the same set of scalers (atdifferent times).

The next set of signals relate to a specific class of optical based eyetracking algorithms. References 7635 through 7640 are control signalsfor a corresponding number of eye tracker camera and illuminationsub-systems. References 7645 through 7650 are data signals back fromthese sub-systems, likely image pixel data to be processed in firmwareby the DSPs.

FIG. 76 also shows eye blink detector inputs 7655 through 7660. Severalsimple schemes are possible, such as the change in IR spectralreflection between the open eye and the skin of the eye lid.

Reference 7665 represents dedicated (e.g., not programmed) control logicand state machines for wherever needed within the headpiece.

Ideally the power for the components in the display capsule could bebrought in externally. So long as multiple interlocks have verified thatthe eye is covered by an EMD in its proper position, power via IR beamscan be safely used to power the EMD wirelessly. References 7670 through7675 are fixed position IR power emitters. These are powered up when theeye tracking system determines that one or more IR power receivers (FIG.78, references 7840, 7845, and 7850) on the EMD are favorably aligned.Preferably an EMD would have a small internal battery (FIG. 78,reference 7825). It would be advantageous if the battery was capable ofpowering the EMD for an entire day and then recharge at night. Anotherpossible power alternative included leaching power from the mechanicalmotion of the eye blinks. Other forms of electromagnetic, magnetic,sonic, or other radiation might be employed.

It is desirable for the headpiece to perform a “cold” reset of an EMDwhen necessary. A special IR input circuit, operating at a specificnarrow frequency and pattern can be hardwired to a cold reset of thecircuitry within an EMD. The IR signal generator that sends such asignal is reference 7680.

A low bandwidth back-channel free space communication of informationfrom the display capsule to the external electronics attached to theheadpiece is also desirable, reference 7685. In normal operation, thedisplay capsule does not have much to communicate back to the rest ofthe system: perhaps “keep alive” pings, input FIFO fill status, capsulebased blink detection, optional accelerometer data, or even very smallcalibration images of the retina. Also, when the CLMD is not being worn,it may reside in a containment case that possibly runs diagnostics. Theback-channel itself can be a short burst low power infrared channel backto the headpiece electronics, but just as with the pixel input channel,other embodiments may use other communication techniques for theback-channel.

Many of the current video encoding formats also carry high fidelityaudio. Such audio data could be passed along with the PCPDS, butseparated out within the headpiece. Binaural audio could be brought outvia a standard mini headphone or earbud jack 7690, but because thesystem in many cases will know the orientation of the head (and thus theears) within the environment, a more sophisticated multi-channel audioto binaural audio conversion could be performed first, perhaps usingindividual HRTF (head related transfer function) data. Feed-backmicrophones in the earbuds would allow for computation of active noisesuppression by the audio portion of the headpiece.

FIG. 77, reference 7700, shows an example headpiece from the back side.Here eye tracking camera nacelles 7710 through 7710 are shown, as wellas the IR power out 7670 through 7675, and the cold reset out 7680.

It is usually desirable that as much electronics, processing, sensing,etc. be located external to the eye mounted display. However withtoday's electronics capability, several essential electronics andprocessing can be combined onto a single chip mounted within the displaycapsule, but outside the optical zone.

FIG. 78, reference 7800, shows an overhead view of the display capsulewith the positions of several discrete components shown. Reference 7805are the eye blink detectors. Reference 7810 is the main EMD control IC(or equivalent technology). Reference 7815 are accelerometers. Reference7820 delineates the apertures for the femto projectors in thisparticular EMD. Reference 7825 shows one possible location outside theoptical aperture for a (relatively) substantial rechargeable battery: atoroid around the outer edge of the display capsule. So long as externalpower is available, a considerably smaller battery would be more thansufficient; its size would likely be smaller than the controller IC.Reference 7830 delineates the optical zone limit for this particularEMD; the complement of this field is the non-optical zone 7835. Notethat just as with any contact lens, the supported optical zone whichdefines limits on field of view of the eye does not have to be as largeas the natural corneal optical zone equivalent field of view. Naturallyas large as possible of optical zone is desirable (and supportable byEMD technologies), but people commonly use contact lenses and glassesthat have limited optical zones. Possible infrared power in cells areshown as references 7840, 7845, and 7850.

FIG. 79 describes much of the internal function and operation of theelectronics within the display capsule at a block diagram level. Digitaldata streams of pseudo cone pixels are captured by light (sent by theheadpiece) to photo-diode 7910 (or some similar mechanism), and thensent to the controller chip 7905 data input section 7930. This datainput section has several responsibilities. First is decoding the datafields from the carrier, e.g. start bits, ECC or other similar datacorrection technique, decrypted data fields, monitoring internal FIFOstatus and re-impedance matching either by increasing or decreasinginternal pixel clock rates, and/or sending data rate run over/understatus to the headpiece via the back-channel 7955, where there is spacefor much larger impedance matching FIFOs. In cases where a data block istoo corrupted for correction, the input block may send a re-send requestfor the entire block to the headpiece.

After correct decoded data has been captured, it is routed to the properinternal FIFOs on the chip 7905; one for each femto projector 7915 onthe EMD. At the correct timing, the pseudo cone pixel data (plus controldata) will be sent to the femto projectors via the pseudo cone pixeloutput 7935.

The control chip has several optional additional monitors of thephysical world. Temperature via the thermocouple 7940, rapid eyemovement via the accelerometers 7945, blink detection via a specialblink detection circuit 7950 (possibly a line of photo-diodes), etc.

One method for positioning a CMD is to dehydrate tear fluid at the edgesof the contact lens when it is first put on the eye. Dehydratedtear-fluid is mostly comprised of sticky mucous, and thus the user's ownnatural body elements are used to create temporary glue. When it is timeto take the CMD off, a small amount of water eye-dropped into the eyeswill re-hydrate the tear fluid “glue,” decoupling the CMD from thecornea for removal. One way for the CMD to de-hydrate a ring of tearfluid is to locally wick the water portion away. These wicks could beturned on and off by the controller chip 7905.

There are many mechanisms to build in high reliability, testability, andreal-time resets of multiple chip based systems. Only a simple examplewill be given here. The “local reset” 7970 is an output of controllerchip 7905. It resets all the internals of the femto projectors, but notthe controller chip itself. It is possible that the femto projectorscould be reset as often as once per frame, or otherwise as needed. Theexternal reset 7975 is a low frequency signal sent by the headpiece to aseparate circuit than the controller chip that allows the headpiece toperform a hard reset of the controller chip if it is not responding orbehaving properly. It is possible that the controller chip could bereset as often as once per eye blink (˜every 3 to 4 seconds), orotherwise as needed.

Finally, a test loop out 7980 and test loop in 7985 on the controllerchip are present to allow the controller chip to test the femtoprojectors during any system test time, which could be as often as everyeye blink. It is also possible that there will be a linear camera chipsomewhere outside the utilized, but inside the generated, optical pathof each femto display that allows for per pseudo cone pixel calibration.

FIG. 80 shows a block diagram of the electronics portion 8000 of a femtodisplay. It includes two chips: a logic chip 8005 with analog outcontrol chip; and a gallium nitride chip 8010 with 128 UV LEDs arrangedin a bar. The logic chip 8005 receives a stream of pseudo cone pixelsfrom one of the outputs of the controller chip 7905. These are storedinto an input FIFO 8020. After an entire new “scan line” of pseudo conepixels have arrived in the input FIFO, the input FIFO transfers inparallel all of the pixels into a second FIFO, the output FIFO 8025.Each digital data value in the output FIFO is attached to an individualdigital to analog converter circuit 8030, which analog outputs are wiredone-to-one to analog inputs of the GaN UV LED chip. Thus the new line ofvalues being transferred to the LEDs cause a new linear pixel array ofUV light intensities to radiate out and reflect off the currentorientation of the oscillating mirror 8120, and then strike the row ofphosphors 8130 that the mirror 8120 is currently aiming at. In this wayan entire frame of pseudo cone pixels is driven into the femtoprojector.

Because the individual logic chips 8005 have so little circuitry, ifmore FIFO space for data over/under run is needed within the CMD, it maymake more sense to add several additional lines of pseudo cone pixels tothe logic chip 8005 rather than n times more storage on the controllerchip 7905, where n is equal to the number of individual femto projectorson the CMD, likely 40+. Also, along with each line of pseudo cone pixeldata, several additional bits of control and state information can beloaded into the logic chips 8005 per line. This allows the controllerchip 7905 to directly set the state machine(s) of the logic chip at will(think of this as “an instruction”).

A sub-circuit reference 8035 to help synchronize the oscillating mirror8120 to the desired frame and sub-frame rate is also present within thelogic chip 8005. This is part of a larger circuit responsible forpowering and controlling the MEMS (or other) mirror 8120.

For completeness, FIG. 80 also shows the local reset 8040, test data in8045, and test data out 8050.

The physical two dimensional cross sectional view of a UV LED bar,oscillating mirror, and phosphor that comprise the light generatingportion of a femto projector for the case of the mirror and UV LED barpositioned to illuminate the phosphor array from behind is shown in FIG.81, reference 8100. The three dimensional perspective view of the sameconfiguration is shown in FIG. 82, reference 8200.

The physical two dimensional cross sectional view of a UV LED bar,oscillating mirror, and phosphor that comprise the light generatingportion of a femto projector in the case of the mirror and UV LED barpositioned to illuminate the phosphor array from in front is shown inFIG. 83, reference 8300. The three dimensional perspective view of thesame configuration is shown in FIG. 84, reference 8400.

Turning now to power for the CMD, a totally internal solution is atoroidal battery that is recharged at night, but this is only possibleif the total power needs of the CMD over a total work day can be met bythe battery technology that can fit into the CMD somewhere outside theoptical zone. Another possibility is using the eye lid blinks to skimsome of the mechanical power to internal electrical power. A smallerbattery and/or a large capacitor would be needed for buffering.

External solutions can be any of many forms of radiated energy:electrical, magnetic, acoustical, IR optical, visible light optical, UVlight optical, etc. Some sufficiently energetic form of light basedpower could be used where the interlocks guarantee that the power beamoriginating from the headpiece will be turned on only when it is knownto a extremely high degree of probability that the power beam will onlyhit the outer surface of the CMD, and will not pass into the eye becausethe CMD will block that frequency range from propagating through to theeye. A simple example would be an infrared power beam 7670 from theheadpiece pointing at a photovoltaic cell 7920 on the surface of theCMD. Completely IR-blocking coatings on later layers of the CMD mightensure that no spill over will enter the eye. If contact with the CMD islost for any reason, the power beam will be cut off until calibratedcontact is re-established.

Many different tests and data can be used in various combinations toensure that the CMD is positioned properly over an eye. One test is tomake sure that the low bandwidth back-channel from the CMD is beingreceived by some portion of the headpiece, and that the data receiveddescribes normal operation. One piece of such backchannel data is“blink” detectors on the CMD. In one embodiment this can basically be afew dozen photo diodes whose data values can be sent back to theheadpiece for interpretation. Proper eye blinks is a good indicationthat the CMD is properly placed. If the CMD contains a square and/orlinear camera, placed outside the functional optical path, but in aposition to view some portion of the retinal surface, then the “retinalprint” seen by the camera(s) can be used as yet another way to validatethe proper positioning of the CMD. Another test is for theheadpiece-based eye tracker 125 to be functioning properly, and checkthat the eye positions and movements are consistent with a properlyplaced CMD.

V.F Systems Aspects for Image Generators and Eye Mounted Displays

Moving now to EMDS systems aspects, when a headpiece is first connectedto an EMDS and image generators, either physically or via free space,one or both sides can insist on digital signature verification beforeproceeding to normal operation.

Next, somewhere in the system, there may be calibration data for theindividual left and right (or just one) CMDs. While such informationcould be stored somewhere in a networked environment, a convenient andlogical place to place it is in some form of persistent storage in theheadpiece. Once a connection is made between the headset and the rest ofthe EMDS, this calibration information can be copied down the link fromthe headpiece to the scaler components 202 through 210, where it islikely to be stored in the attached memory sub-system. This calibrationinformation can be used to construct the sequential pseudo cone pixeldescriptor list that is assessed during the variable resolutionre-scaling operation.

There are many different methods for implementing head trackers, but aparticular one will be used here as an example. Assume that infra-red(IR) LEDs are mounted on the outside of the headpiece, and are turned onbriefly at a known set of times. The rest of the headtracker, thetracker frame 230, would contain three or more one dimensional or twodimensional infrared cameras. The sub-pixel accurate (via varioustechniques) location of the infrared LEDs captured by the cameras can bedirectly manipulated computationally to give an accurate position andorientation of the headpiece, and thus the position of human user's 110eyes. To perform this task, there should be tight timing synchronizationbetween the transmitters (IR LEDS) and the receivers (1D or 2D IRcameras) in the tracker frame 230. The tracker frame should also sendthe image data captured to a computational unit that can transform itinto viewing matrices for image generators and matrix transforms formapping the virtual screen to the EMDS. This computation could beperformed anywhere within the system, but a good placement would be theheadpiece that already will have a computational infrastructure forextracting eye orientation data. Note that the direction of informationflow is from the scalers to the headpiece.

There are many different methods for implementing eye trackers, but forsimplicity a particular example will be used here. In these cases, acontact lens display has special marks printed and/or embossed on ornear its surface. These marks are illuminated by timed flashes of lightfrom portions of the headpiece. Also on the headpiece are a number oflinear or array cameras (likely infrared) that capture the interactionof the illumination bursts with the patterns. These cameras areadvantageously placed as near the eye as possible. In this example, theyare placed all around the inside rims of a pair of eyeglasses that formpart of the headpiece. This way, no matter what direction an eye islooking, there will be several cameras able to obtain a good image ofthe pattern.

Because the illumination and the cameras are in this case part of theheadpiece, it is advantageous to have the image processing performed onthe camera outputs to determine the orientation of the eyes. Thiscomputation is simple enough that a custom image processor design is notneeded. Existing DSP IP cores should be able to handle this job, and canalso be handed the data from the head tracker cameras.

With the same DSP cores computing both the head and the eye trackingdata, they are advantageously positioned to compute the transforms andother per-frame data that the scalers use to process the next frame, orin parallel frames, of video data. This information flow is from theheadpiece to each scaler individually, as different virtual screens canuse different data. As both the head and eye-tracking may be takingplace at a higher rate than the video rate(s), the data for the scalerswould be averaged (or more complexly) over several sub-frames, and onlysent on to the scalers where the time was just before they need to startprocessing a new frame of data. Once they start, this completes thecycle.

V.G Meta-Window Systems for Eye Mounted Displays

Now consider how to configure the position, orientation, size, andcurvature of the (multiple) virtual display image(s). Certainly one wayis for the EMDS to come with a small controller to allow individuals toset such parameters, similar to how CRTs had controls for the horizontaland vertical height, the horizontal and vertical size, etc., but settingup objects in three dimensions literally adds another dimension to theproblem.

A more likely solution is for an application running on one of thecomputers controlling one or more image generators to have a GUI to letvirtual displays be placed, orientated, and sized; and curvatureparameters set if that option is available. Most modern window systemsallow for some number (at least 8) of separate image generators tobecome the “tiled” portions of what is otherwise a single larger windowworkspace. Moving the cursor off to one side of a display causes it toappear on the physically neighboring display, if there is one there.This covers two of the more common uses of a single computer with anEMDS: n×m image generator separate video outputs form either a singlelarge flat window in space, or a single cylindrically curved window. Itis usually important for the EMDS to know when two window edges areintended to seamlessly abut versus one being to the rear, or front, ofthe other. Such virtual window configurations preferably are persistent,e.g. do not require the user to set them over again every time thecomputer(s) are re-booted. This can be addressed by having theapplication on a computer that handled the creation of the virtualscreen placement parameters insert a “window system start-up time” jobthat will re-send the configuration information whenever the windowsystem is booted. Another option would be to write the virtual screenparameter information into electronically alterable storage within theEMDS. It only need be changed when the configuration application is runagain.

The conventional method to support multiple computers running at thesame time in a single display is to use a KVM: Keyboard, Video, andMouse switcher. This is a box that for example, has one USB keyboard andone USB mouse input, as well as one video output (in some format, analogor digital), but has n USB keyboard and mice outputs, and n videoinputs. The scaler component of an EMDS effectively already performs amore sophisticated control of n video inputs. What is left is control ofkeyboard and mice. If two USB inputs and two USB outputs are added toeach scaler black box (or multiples for black boxes that support morethan one video in), then the scalers can perform a conventional job as aKM (keyboard mouse) switch.

Conventional KVMs allow the user to dynamically specify which of the upto n computers is currently active for keyboard and mouse by means of anadditional multiple button interface device. It would be preferable toavoid adding such additional physical user interface devices. Onepossible solution is to allow the software program that is dynamicallycontrolling the virtual displays to also dynamically control thekeyboard and mouse focus. There are other alternatives: a rapid double“wink” in one eye of the user could change the keyboard and mouse focusto the computer controlling the virtual display that the user iscurrently looking directly at (e.g., use they eye tracking and blinktracking data).

With respect to minimizing a virtual screen, rather than collapsing thescreen to a label on the top or bottom menu bar; it is possible tocollapse it to a “flat” video image within the EMDS display space.Because such “collapsed” video streams are below any active windows,there is (usually) scaler computational bandwidth to include (a perhapsfrozen video image contents) display of these “stubby” virtual screens,perhaps with a text tag associated with it. This “tag” part could be thesame as current window systems. A user control of some sort would allow“un-closing” of the video window at a future point in time. They wouldthen revert to a “normal” virtual screen.

V.H Advantages of Eye Mounted Display Systems

The possible advantages of an eye mounted display system are numerous.One possible advantage is that keeping a display made up of variableresolution display elements coupled close to, or locked to, the variableresolution of the human eye's retinal receptive field centers, meansthat a device that meets or exceeds the resolution and field of viewrequirement of the human visual system can potentially be built.

In addition, just as one uses the same pair of glasses while at work,home, or other outside activities, another possible advantage of eyemounted display systems is that the same pair of eye mounted displayscan be worn and thus replace many fixed displays at these locations.Thus even if an eye mounted display system costs more than anyparticular display, to be economical, it only has to cost less than allthe other fixed displays it replaces.

A third potential advantage of eye mounted display systems is thatbecause eye mounted display systems are inherently small and low inpower consumption, they may be able to solve the display size andresolution limitations of current small portable electronic devices:cell phones, PDAs, handheld games, small still and video cameras, etc.In addition, the approach described here for eye mounted display systemsis compatible with existing video display standards, and has thepossible advantage that it can put more than one video input into thelarger perceptual display space, without requiring the video sources tocommunicate with each other.

Another potential advantage is that for the specialized market wherehead mounted displays are used; an eye mounted display system providesorders of magnitude more perceptible display pixels, much lower weightand bulk, etc. With the combination of large field of view, high spatialresolution, integral head-tracking (on some models), see-throughcapabilities, and potentially low cost, the markets for immersivedisplays can expand to significant sections of the gaming and some ofthe other entertainment markets, while better serving the existingmarkets for head mounted displays in scientific visualization, virtualprototyping, simulators, etc.

Yet another possible advantage is because it is fairly natural toconstruct eye mounted displays that have similar variations inresolution as does the human eye, orders of magnitude fewer displayelements (“pixels”) can be used on a display fixed to the eye than fordisplays that do not know where the eye is looking, and thus mustprovide uniformly high resolution over the entire field of the displayor for displays that cannot assume that only one human 110 observer ispresent and again thus must provide uniformly high resolution over theentire field of the display. As an example, an eye mounted display withonly 400,000 physical pixels can produce imagery that an externaldisplay may need 100 million or more pixels to equal (a factor of 200times less pixels). In principle, a variable resolution display alsoallows image generation or capture devices, whether computer graphicssystems, high resolution image playback systems, still or video camerasystems, etc., to only compute, decompress, transmit, or capture (forcameras) orders of magnitude fewer pixels than would be required for noneye resolution coupled systems.

Eye mounted displays also require vastly fewer photons compared toexisting displays and, therefore, vastly lower power also. Eye mounteddisplays have several properties that most external display technologiescannot easily take advantage of. Because the display is coupled in spacerelatively close to the rotations of the eye, only the amount of lightthat actually will enter the eye (through the pupil) need be produced.These savings are substantial. For an eye mounted display to produce theequitant retinal illumination as a 2,000 lumen video projector viewedfrom 8 feet away, the eye mounted display need only produce one onethousandth or less of a lumen. This is a factor of one million timesfewer photons (both eyes).

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of the methodand apparatus of the present invention disclosed herein withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. An eye mounted display system comprising: anintraocular display capsule configured to be placed inside a user's eye;and a femto projector contained in the intraocular display capsule, thefemto projector projecting a plurality of pixels onto a retina of theuser's eye, thereby forming a visual sensation of an image comprisingthe pixels, the femto projector comprising: an array of addressabledisplay pixels, and display optics projecting light from the displaypixels to a portion of the retina corresponding to the image displayedby the femto projector, where the portion of the retina is fixed as theuser's eye rotates in its socket and thereby forming the visualsensation of the image comprising the pixels.
 2. The eye mounted displaysystem of claim 1 where the intraocular display capsule is configured tobe placed within an anterior chamber of the user's eye between a corneaand a lens of the user's eye.
 3. The eye mounted display system of claim1 where the intraocular display capsule is configured to be placed on afront surface of a lens of the user's eye.
 4. The eye mounted displaysystem of claim 1 where the intraocular display capsule is configured tobe placed inside a lens of the user's eye.
 5. The eye mounted displaysystem of claim 1 where the intraocular display capsule is configured toreplace a lens of the user's eye.
 6. The eye mounted display system ofclaim 1 where the intraocular display capsule is configured to be placedon a rear surface of a lens of the user's eye.
 7. The eye mounteddisplay system of claim 1 where the projected pixels having differentsizes at the retina.
 8. The eye mounted display system of claim 7 wherethe sizes of the projected pixels at the retina vary in part as afunction of a size of retinal receptive fields at the retina.
 9. The eyemounted display system of claim 7 where the pixels projected to a centerof a fovea of the eye have a smallest size among all the projectedpixels.
 10. The eye mounted display system of claim 1 where the sizes ofthe projected pixels at the retina have a resolution that is at least ashigh as a native resolution of the eye.
 11. The eye mounted displaysystem of claim 1 where the array of addressable display pixels includedisplay pixels of different colors, whereby the femto projector projectsa color image onto the retina.
 12. The eye mounted display system ofclaim 1 where the femto projector comprises a plurality of sub-displays,each sub-display comprising: a plurality of display pixels displaying adifferent portion of the image for each sub-display, and display opticsprojecting light from the display pixels to a portion of the retinacorresponding to the portion of the image displayed by that sub-display,different sub-displays projecting light to different portions of theretina, and the sub-displays in aggregate projecting light to portionsof the retina that in aggregate form the image displayed by the femtoprojector.
 13. The eye mounted display system of claim 1 where the arrayof addressable display pixels is a hexagonal array.
 14. The eye mounteddisplay system of claim 1 further comprising: a logic chip coupled to anLED chip, the LED chip containing the array of addressable displaypixels, the logic chip receiving data specifying the pixels projected bythe femto projector and producing analog drive signals for the LED chip.15. The eye mounted display system of claim 1 further comprising: a datareceiver device contained in the intraocular display capsule forwirelessly receiving data that specifies the pixels projected by thefemto projector.
 16. The eye mounted display system of claim 1 furthercomprising: a power receiver device contained in the intraocular displaycapsule that wirelessly receives power for the femto projector.
 17. Theeye mounted display system of claim 1 further comprising: anaccelerometer contained in the intraocular display capsule for measuringmovement of the eye.
 18. The eye mounted display system of claim 1further comprising: a device contained in the intraocular displaycapsule for wirelessly receiving data and/or power for the femtoprojector using a water penetrating, mid-infrared frequency.
 19. The eyemounted display system of claim 1 further comprising an eye tracker thattracks an orientation and/or position of the eye, where the imageprojected by the femto projector is a function of the orientation and/orposition of the eye.
 20. The eye mounted display system of claim 1further comprising a controller chip that transmits to the femtoprojector data specifying the pixels projected by the femto projector.